ARCHIVED - Evaluation of NRCan’s Transportation S&T Sub-sub Activity

Information Archived on the Web

Information identified as archived on the Web is for reference, research or recordkeeping purposes. It has not been altered or updated after the date of archiving. Web pages that are archived on the Web are not subject to the Government of Canada Web Standards. As per the Communications Policy of the Government of Canada, you can request alternate formats. Please "contact us" to request a format other than those available.


Table of Contents


EXECUTIVE SUMMARY

Purpose
This report summarizes the findings of an evaluation conducted on the Transportation Science and Technology (S&T) Programs of Natural Resources Canada’s Energy Sector.1 The evaluation covers approximately $93.8M of NRCan funding for the period 2002-03 to 2006-07.

Context
Transportation plays a vital role in Canada. In 2006, transportation-related demand accounted for 12.2 percent of Canada's gross domestic product (GDP). The Canadian economy relies heavily on transportation for moving goods and services, making significant demands on the transportation sector.

Transportation is the second largest source of greenhouse gases (GHGs) in Canada (behind energy production), accounting for 27 percent of Canada’s emissions growth from 1990 to 2005. Over this timeframe, transportation CO2 emissions increased by 33 percent, from 150 megatonnes (Mt) to 200 Mt.

Background
The Transportation Energy S&T Sub-sub Activity consists of six programs that are multi-sectoral and interdepartmental and are aimed at developing clean and efficient energy technologies for the transportation sector. The program’s activities encompass: basic to applied research and development (R&D); support for the development of codes, standards and policy; development and demonstration of technologies; and process development.

The programs are intended to achieve environmental and economic impacts through a series of collaborative relationships among their partners (public and private S&T performers). The S&T performers include NRCan’s CANMET Energy Technology Centres, other public sector laboratories, academic and private sector laboratories/technical groups, as well as implementer and user communities and various standards-setting groups. These organizations perform research, development and demonstration (RD&D) functions.

Six programs and elements of a seventh comprise this Sub-sub Activity:

  • Advanced Fuels and Transportation Emissions Reduction (AFTER) Program;
  • Canadian Lightweight Material Research Initiative (CLiMRI) Program;
  • Particulate Matter (PM) Program;
  • Technology and Innovation (T&I) Transportation Program;
  • Hydrogen Energy Economy (HEE);
  • Canadian Transportation Fuel Cell Alliance (CTFCA); and
  • some aspects of Technology Early Action Measures (TEAM).2

Three of these programs (AFTER; CLiMRI; PM) are managed by NRCan’s Office of Energy Research and Development (OERD) as part of the Program of Energy Research and Development (PERD). The HEE Program is jointly delivered by PERD and NRCan’s CanmetENERGY Technology Centre-Ottawa (CETC-O). The Canadian Transportation Fuel Cell Alliance was managed by CanmetENERGY Technology Centre-Ottawa (CETC-O). The T&I Transportation Program was managed by OERD. Each of the programs evaluated functions as a separate entity although there are linkages between the CTFCA and HEE Programs.

Evaluation Issues and Methodology
This study examined issues related to the programs’ relevance/rationale, results and success and cost-effectiveness. The evaluation methodologies included:

Document Review – this included over 200 documents which encompassed each program’s documentation, plans and performance reports. In addition, Canadian federal S&T policy documents (e.g., the Budget Speech, Speech from the Throne and substantiating analyses, legislation, etc.) as well as related publications of other countries and federal departments, reports to Parliament, and technical publications were reviewed.

Interviews – 61 interviews were completed. Interviewees included program managers, project leaders, industry stakeholders and partners. They were typically selected from members of the management and advisory committees involved in each program. The following table describes the breakdown by the representative groups:

Evaluation Interviews
NRCan OGD Industry University Provinces International Total
14 21 18 5 2 1 61

Case Studies – Twenty five in-depth case studies were completed. The case studies were conducted across a sample of projects to acquire more detailed knowledge of outputs and outcomes and are one of the major data sources for addressing the success of the programs. They involved a review of data and documentation, as well as 61 interviews with project deliverers and stakeholders. The following table describes the breakdown of the case study interviews:

Case Study Interviews
NRCan OGD Industry University Provinces International Total
18 16 24 1 0 2 61

Project Reviews – To supplement the document review and the case studies, a detailed review of 19 projects was undertaken. This involved an examination of project documentation to provide detailed information about the relevance and results of the selected projects.

Evaluation limitations – The Transportation S&T investment system is complex, involving many technical program areas and funding initiatives that are on different funding cycles. These programs involve a large number of federal departments, industry, universities, and other stakeholders. They cover a range of activities and objectives and exist as a loose cluster within the Transportation S&T theme of NRCan’s PAA. During the time period covered by this evaluation there was no strategy or policy framework that encompassed the programs. Since there was no up-to-date policy or strategy that outlined federal transportation S&T objectives and priorities linking the programs together, the evaluation used the multiple policy statements on energy S&T as the basis for the examination of the Relevance and Success evaluation issues. (Since the end of the evaluation period, work has been undertaken to develop a strategy.)

Findings

Relevance/Rationale
All of the programs were found to be relevant to federal priorities, NRCan priorities, and the needs and priorities of stakeholders. The priorities addressed by the six programs are related to federal and departmental priorities as they are intended to:

  • reduce emissions of pollutants, including greenhouse gases;
  • improve air quality and health by supporting technology and policy relevant-research and development (R&D);
  • ensure the efficient use of natural resources;
  • support private sector competitiveness; and
  • increase knowledge through S&T.

The six programs are relevant to NRCan’s mandate in that they:

  • are concerned with sustainable development and the wise use of Canada’s resources (both energy and mineral);
  • involve natural resource-related R&D focussed on energy;
  • develop capacity among the stakeholders;
  • develop knowledge and technology and transfer it through RD&D partnerships;
  • provide knowledge needed to support policy development; and
  • are oriented towards improving the competitiveness of Canadian industry.

The PM Program, while relevant to NRCan, was found to have a different relationship to NRCan’s mandate than the other NRCan transportation S&T programs. The PM Program is unique as its activities are of a more policy-focused, less-applied (i.e., fundamental research) nature than the objectives articulated in NRCan’s Energy Science and Technology Companion Document and the ecoEnergy Technology Initiative which are focussed more on technology development. The PM Program’s goal is, “to strengthen the scientific basis for policy and regulatory decisions effecting transportation-related emissions of particulate matter and its precursors.” The types of tools and knowledge that are being developed by this Program are intended to play a role in providing sound science for policy decisions.

As a result of its focus on providing scientific data to support policy and regulatory decisions related to air quality, the Program is directly related to the priorities of Environment Canada which has a legislated duty under the Canadian Environmental Protection Act, 1999 (CEPA 1999) to conduct research related to particulate matter. In order to implement its mandate, Environment Canada’s Air Quality Research Branch (AQRB)3 conducts research on particulate matter with the goal of developing the tools and knowledge to evaluate possible fuel and other transportation-related standards (e.g., emissions and fuel efficiency), which may be needed to meet future particulate matter air quality standards. The PM Program is also directly aligned with Health Canada’s legislated duty to conduct research relating to the role of substances, such as exhaust emissions, in illnesses.4

The PM Program remains appropriate to NRCan’s PERD, since PERD is an interdepartmental R&D funding program whose objective is, “to provide the continuing science and technology necessary for Canada to move towards a sustainable energy future.” Although the Program has strong linkages with Environment Canada, interviewees reported that it is appropriate for NRCan to be responsible for the Program because NRCan’s mandate relates to energy and the use of energy, while Environment Canada focuses more on environmental protection. Interviewees commented that they view NRCan as the leader in horizontal R&D and that without the involvement of PERD a silo effect might develop and the research would lose its links with health effects.

In focussing on their specific areas of interest, the six programs are relevant to more general national needs and priorities such as the efficient use of resources and improving private sector competitiveness. Strong understanding of the technological, environmental and human health impacts of transportation technologies operating under Canadian conditions is important, and therefore relevant, to supporting technology development as well as informing Canadian policy and regulatory decisions.

The six programs were found to be relevant to addressing needs of the transportation sector:

  • to reduce emissions of pollutants, including GHGs;
  • to improve air quality by supporting technology and policy relevant R&D; and
  • to increase knowledge through conducting S&T in specific areas.

Two of the six programs, the CTFCA and the HEE, addressed infrastructure-related needs associated with hydrogen fuels. These needs are strongly influenced by a "chicken and egg" syndrome that forms a significant barrier to the introduction of new fuels and related technologies, arising from the interaction of three considerations:

  • reluctance of vehicle manufacturers to produce vehicles that operate on new fuels in the absence of a fuel delivery infrastructure to deliver these new fuels, combined with;
  • reluctance of consumers to purchase such vehicles given the lack of a fuel delivery infrastructure; and
  • reluctance of energy firms to construct a new fuel delivery infrastructure in the absence of sufficient demand, (i.e., a critical mass of vehicles operating on the new fuels already in use).

Four of the six programs were found to be relevant to the transportation sector in that they addressed the need for policy instruments for new fuels such as hydrogen (HEE together with the CTFCA) and for existing technologies (AFTER and the PM Program). This need arises from the fact that the introduction of new fuels and technologies can be delayed in cases where appropriate policy instruments are lacking, such as guidelines, regulations, and installation codes for basic safety and operability requirements (e.g., storage, installation, and delivery) of the new fuel and/or technology.

The programs were found to be relevant to another transportation sector need to comply with existing and proposed policies. This need exists in government and industry segments of the transportation sector, since both require policy and technology relevant knowledge. The transportation industry also needs improvements in current transportation technologies to comply with government policy requirements. Examples of specific areas addressed include:

  • AFTER: particulate matter filters and sensor technologies;
  • CLiMRI: advanced forming and fabricating technologies for advanced lightweight materials and vehicle components;
  • HEE: technologies improving fuel cell and fuel cell engines including storage;
  • CTFCA: evaluating the performance of hydrogen technologies;
  • PM: tools and methods to analyze PM; and
  • Technology and Innovation (T&I) Transportation Program: supporting CLiMRI, PM and AFTER.

All of the programs advanced the state of knowledge in their particular areas of focus. In the policy area, the knowledge resulted in better understanding of the causes and effects of areas being regulated (e.g., PM, AFTER, HEE) and supported development of models with potential for policy development application (e.g., AFTER, PM).

In the technology area, the knowledge developed by the programs was fundamentally important for technology development. For example, better understanding of the characteristics of lightweight materials is essential to their use in vehicle construction and manufacturing.

Finally, in order to develop technology and knowledge (e.g., to meet compliance needs), industry and academia require ready access to federal laboratory facilities, knowledge, and expertise to address technological and process-related challenges. All of the programs met this need by making NRCan laboratory facilities, knowledge, and/or expertise not found elsewhere available to program stakeholders.

Results and Success The programs produced research outputs such as better information, as well as improved fuels, materials, and technologies. With respect to actual impacts for Canadian society, the programs supported new manufactured products, infrastructure and policy. These impacts are influenced by external factors and players, but the evaluation evidence shows that the programs are key determinants of some of these impacts. The impacts include better air quality, safer equipment, reduced fuel consumption as well as economic impacts such as revenues for Canadian firms and savings for consumers and transportation companies.

In terms of attribution, the programs are not the sole contributors to these impacts. However, at least some of the projects would not have been carried out in the absence of NRCan’s participation. Others would have proceeded with reduced scopes. This indicates that the NRCan funding, facilities and expertise played a substantive role in the achievement of these results.

The following describes general Success findings for each program:

AFTER: The Program has made progress towards achieving its objectives in that it has generated knowledge in all of its four R&D activity areas.5 The majority of outcomes are related to advancing understanding. The research resulted in more than forty-five journal publications to date, and produced forty-four conference and technical meeting presentations. With respect to technology development, the Program developed three pre-commercial sensor prototypes that monitor engine performance to control emissions that are currently being tested by industry. A patent application has been filed for one of these technologies and another has had its patent application approved. The granted patent has been licensed, an instrument developed, and in excess of 12 sales recorded, primarily to engine manufacturers.

Overall, interviewees reported that the Program is progressing towards achieving its objective and that work on fuels, engine technologies and health impacts are contributing to the field of knowledge in the areas studied. Several interviewees emphasized the importance of the Program’s work in comparing oil sands derived fuels to conventional sources, and an oil industry representative commented that the fuel chemistry work is adding to the knowledge base in this area. Interviewees reported that as a result of AFTER’s work they have a better understanding of how fuels interact with engines and of the health risks involved.

PM Program: The Program generated knowledge related to the role of transportation in the production of particulate matter, the evolution of particulate matter in the atmosphere and the human health impacts of particulate matter. The Program’s outputs include 25 published papers, 11 conference presentations, and model development and testing. The Program developed and enhanced tools and methods to analyze PM and to produce information that could inform upcoming policy and regulatory development processes.

The Program’s main success may be its horizontal view that examines the whole spectrum from tailpipe emissions to cardio-respiratory impacts, thus generating knowledge and awareness on potential health effects area and ensuring that they are not under-emphasized.

The Program’s research results could, in the near future, influence the policy and regulatory communities. However, at this point in time, there is limited evidence that the knowledge has been used to strengthen policy and regulatory decisions. In a general sense, it was felt that information from the PM Program and other programs contributed to the creation of the Canada Wide Standards, Ozone Standard, PM Annex and was used in negotiations with the US on particulate matter from transportation.

The Program influenced the engagement of stakeholders and collaborative networks to further research and development. However, interviewees noted the apparent disconnect between policy and regulatory groups and research and development. To date, the effort that the Program has been able to exert in this area relies heavily on the initiative of key individuals within the Program.

CLiMRI: The Program generated knowledge related to reducing GHGs through weight reduction and improved vehicle efficiency by building knowledge of lightweight materials that can be used to make efficient designs and components. Advances were made in the areas of aluminum, magnesium, ultra-high strength steel (UHSS), and titanium. CLiMRI also developed knowledge (e.g., processes, technology pathways) which, if implemented by industry, would decrease operating costs of using these lightweight materials. CLiMRI’s expertise is well-recognised in the area of designing and processing metals technologies.

It should be noted that the reduction of GHGs will only be realized by commercialization of this knowledge. Commercialization depends on a number of factors, including involvement of industry partners in the R&D itself to facilitate technology transfer. Identification of partners is usually part of CLIMRI project planning with the exception of the brake rotors project where no partner was identified. With this one exception, CLiMRI has been successful in developing domestic and multilateral partnerships.

T&I Transportation: The Program developed knowledge in three of its four theme areas (i.e., Vehicle Materials and Design Efficiencies; Advanced, Efficient Powertrains and Fuels; and Support to Policy Development and Integration)6, which expanded the knowledge base of transportation’s contribution to climate change. The knowledge base includes characterizing GHG emissions from vehicles with diesel engines, off-road vehicles, and vehicles with advanced emission control technologies. Findings from this research suggest that current emission factors7 overestimate the GHG emissions from light duty vehicles in Canada. The Program has provided data to Environment Canada’s emissions inventory group so that the emission factors may be updated.

The Program also generated a substantial amount of knowledge with respect to black carbon’s (BC) role in climate change, such as scientific advances related to BC measurement, databases/inventories and models. In the medium and longer-term, it is anticipated that knowledge generated by this project will contribute to improved policy development and assessment.

With respect to advancing the development and implementation of promising new technologies the Program generated the following outputs: two new high temperature aluminum alloys that are expected to tolerate the harsh conditions of a diesel engine; initiation of an ambitious and major multilateral technology project to develop a magnesium automobile front-end (jointly funded with the CLiMRI Program); development of a composite material with nano-reinforcement which was applied to aluminum brake rotors (jointly funded with CLiMRI); and development of novel electrolytes that are both safer and less expensive compared to existing electrolytes, which meet many of the performance characteristics for usage in lithium ion batteries.

A range of factors were identified by interviewees as having influenced the performance of the T&I Transportation Program. These included the Program’s short lifespan (2003-04 to 2007-08) and lack of preparation time to launch it. The former created issues with respect to hiring research scientists and the latter created a one-year delay in the Program becoming operational. The Program Leader changed three times in five years, which did not support the Program’s stability. Additionally, the Program’s reliance on a request for proposal approach to build its research program was described as having produced an unequal distribution of funds among the Program themes (i.e., two of the four research themes were relatively inactive).

HEE: The HEE Program was successful in advancing the state of knowledge by conducting R&D to fill knowledge gaps in areas identified as priorities by governments and industry.

Some of the key successes are described below:
  • supported the development of codes and standards such as the Canadian Hydrogen Installation Code (CHIC), which made Canada the first country in the world with a hydrogen installation code;
  • advanced the development of fuel cell technologies by addressing knowledge gaps;
  • contributed to the development of the world’s first compressed hydrogen cylinder to demonstrate the ability to safely store hydrogen at 700-bar (10,000 pounds per square inch); and
  • participated on behalf of Canada in two International Energy Agency Implementing Agreements related to hydrogen and fuel cells, which ensured Canadian input into any internationally recognized codes and standards that are developed.

CTFCA: The CTFCA successfully demonstrated and evaluated options for hydrogen fuelling stations and hydrogen-fuelled vehicles in Canada. This was accomplished by showcasing refuelling demonstration projects; evaluating various hydrogen-fuelled light, medium and heavy duty vehicles; and delivering the national supporting framework needed to enable the development of the fuelling infrastructure such as technical codes and standards, training, certification and safety. Monitoring the resulting greenhouse gas emissions from projects was listed as a sub-objective of the Program, and the Program is now (June 2009) compiling and analysing that data. The data were not available at the time of this evaluation. It should be noted that demonstration projects were selected in part because of their potential impact on reducing GHG emissions based on the hydrogen pathways that were modeled by NRCan’s GHGenius model.

Some of the CTFCA’s key successes are described below:

  • contributed to the development of 11 permanent (and four in development phase) hydrogen fuelling stations;
  • successfully demonstrated a total of 60 fuel-cell powered vehicles, some of which were dual-fuelled (by gasoline or diesel- as well as fuel cells);
  • established some of the basic national policy instruments (e.g., guidelines, regulations, and installation codes for basic safety and operability requirements such as storage, installation, and delivery) needed to support implementation of hydrogen as a power currency and contributed to international work in the same area; and
  • provided advice to industry and government on the most viable hydrogen pathways.

Cost Effectiveness
Overall, the Transportation Sub-sub Activity was delivered in a cost-effective manner. The programs were generally appropriately structured and efficient for generating research results. The planning, review and reallocation processes for projects ensured that they fit under the overall direction and objectives of each program. It should be noted that because of the nature of research and development, it is difficult to demonstrate cost-effectiveness in relation to the immediate return on investment (ROI). Results from research and development take place over a longer timeframe than ROI and require a long-term commitment. Some examples of economic impacts are presented in the section below.

With respect to leveraging funds from other sources, the programs leveraged $29.1M from other federal government departments and $75.8M from non-federal government sources. In comparison, the total NRCan funding of the Transportation S&T Sub-sub Activity from 2002-03 to 2006-07 was approximately $93.8M, which includes PERD, CCTII and A-base funding. The leveraging ratio of Government of Canada to non-Government of Canada funding was 1.6 to 1 for the Transportation S&T Sub-sub Activity.

Similarities between the T&I Transportation Program, which ended in March 2008, and the PERD Transportation programs, led to some duplication of effort for S&T performers with respect to participating on various committees, submission of proposals, project monitoring and reporting. Similarities also led to a number of projects that were jointly funded by T&I and PERD. In some cases, the T&I Transportation Program reported the same results and funding for the projects reported by the Particulate Matter, CLiMRI, and AFTER programs.

Potential for Economic Impacts and GHG Reductions
Three key factors serve as a context for the potential economic impacts and GHG reductions of the RD&D projects:

  1. GHG reductions and economic impacts can only be achieved to the extent that the technologies and knowledge are commercialized by industry. The programs developed knowledge and expertise, tested and showcased technologies. They were not intended to commercialise the research and development and demonstration (RD&D) results. The programs rely on publications and partnerships with the automotive supply chain for technology transfer.
  2. When the programs are successful in developing new technology, commercialization requires substantial time and resources. Where proven technologies exist, the lead time for introducing them is on the order of 5-10 years and may involve hundreds of millions of dollars of investment.
  3. The nature of some of the results (i.e., technical capacity and product development) can be difficult to measure. Difficulties linking results to specific projects – combined with third party delivery, makes attribution of economic and GHG impacts difficult.

Examples of actual and potential impacts of the programs are:

GHG reductions:

  • CLiMRI: Lightweight Vehicle Body Architecture Project: 6,000 metric tonnes by 2014 (if implemented in 2012).
  • Transportation T&I: Reduction in Fuel Consumption via Reduction of Aerodynamic Drag: As of 2008, 110 truck trailers have adopted these components, which will – in the most pessimistic scenario – result in 800,000 litres of fuel and 2,160 metric tonnes carbon dioxide equivalent (CO2 eq.)9 in savings over five years. If 8,000 truck trailers (five percent of Canadian fleet) adopted the components the likely reductions would be 157,000 metric tonnes over five-years.
  • CTFCA: The Vancouver Fuel Cell Vehicle Program: 18,500 metric tonnes in actual savings. (Timeframe: 2005 to 2008, with 168,000 km travelled.)
  • TEAM: The TEAM program developed the System for Measurement & Reporting of Technologies (SMART) process to determine the GHG emission reductions achieved in demonstration projects and applied this methodology to the transportation demonstration projects funded during the period encompassed by this evaluation. The Program also developed a sector-specific protocol called "Fuel Cell Transportation Sector Specific protocol" which can be generally applied to such projects. The protocol is the basis of a new, international standard (#14064 part 2),10 which now applies to any GHG emission reducing project.

Total: 20,660 metric tonnes of CO2 eq in actual savings over a five-year period.

Cost savings to Canadians from reduced fuel consumption:

  • CLiMRI: Lightweight Vehicle Body Architecture Project: $2.6M (if implemented).
  • Transportation T&I: Reduction in Fuel Consumption via Reduction of Aerodynamic Drag: To date, 110 truck trailers have adopted these components, which will – in the most pessimistic scenario – result in $0.8M in savings over five years. If 8,000 truck trailers (five percent of Canadian fleet) adopted the components the likely cost savings would be $58M over a five-year period.

Total: $0.8M in actual cost savings over a five-year period.

Revenues and sales generated:

  • CLiMRI: Galvanic Corrosion: $400K in sales for two companies, from 2005-2007.
  • CTFCA: The Hydrogen Highway Project: $3.5M in sales (estimated sales in 2008 and 2009 could total $15M).
  • T&I - TEAM: Home natural gas refuelling appliance: $6M in annual sales, from 2005 to 2008.

Total: $9.9M in actual sales, from 2005 to 2008.

Collaborative Networks
Overall, the programs were generally effective in bringing together a diverse group of organisations to further RD&D goals. This involved interdepartmental participation in conducting projects, managing the programs, and sharing information at the project, program and departmental level. Technical and financial collaboration within projects were established among other government departments, industry, universities, provinces, and organizations in other countries. In general, these links were found to have a beneficial impact. However, in the case of one program (CLiMRI), discussion of technical needs was reported as having been limited by the presence of competitors.

From 2001-02 to 2007-08, OERD provided approximately $72.8M ($57.2M in PERD and $15.6 in CCTII) funds to six federal government departments/agencies for transportation S&T. NRCan’s CANMET received approximately $46.9M (64 percent of PERD and 67 percent of CCTII) of the total funding provided by OERD. Funding multiple departments allows for a multi-disciplinary approach to address the research issues which assists in developing working relationships and reduces silos.

Although the evaluation found that collaboration among departments and researchers was working well, there is limited information on the level of engagement of industry and the policy and regulatory communities. Achievement of the programs’ objectives is dependent on the uptake and implementation of the research results by these groups.

Performance Reporting
The six programs generally did not report on their performance frameworks. Annual reports (one program did not produce annual reports for two years) tended to convey information regarding the technical achievements of individual projects and often did not make the linkages to the desired outcomes. This made it difficult to use the annual reports to understand progress towards achieving program objectives. This was not the case for project reports produced by private sector project proponents. The private sector reports contained readily understandable information on the results that projects are intended to achieve, the significance of the results in terms of the program’s objective, and the timelines and resources involved.

Interviewees reported that although meetings were effective in sharing information, program documentation such as annual reports were not always effective in communicating results. In the case of at least two programs, interviewees described the benefits of results-based management (RBM), such as helping researchers focus on results.

Financial information generally varied within and across the six programs with respect to quality and availability; and expenditure data were seldom reported.

Some specific reporting issues include:

  1. Actual expenditures vs. budgets: The annual reports do not contain information on actual expenditures.
  2. Data Inconsistencies: The funding data in the programs’ annual reports were not consistent with the overall funding levels per program reported by OERD and the CTFCA. In addition, the funding data reported in the annual reports were not always consistent with the funding reported in their project financial databases. As a result, it was difficult to determine which information was accurate.
  3. Labelling: Funding sources were sometimes not clearly identified and were lumped into one category as “Other” or “Other Government”.
  4. In some cases the same project results were repeated for several years. For example, a project from 2003-04 reported the same information in the annual report of 2006-07 as it did in 2003-04.
  5. Some of the multi-year projects never reported an update after one year. Therefore, it was difficult to determine whether the projects were successfully completed or terminated.

1 Based on the Clean Transportation Energy Sub-sub Activity 2.1.4.2 of the department’s 2008-09 Program Activity Architecture.
2TEAM was not evaluated as the majority of TEAM demonstration projects do not fit within the Transportation Energy S&T Sub-sub Activity. However, four transportation related projects were identified and assessed as part of the case studies for this evaluation.
3http://www.msc-smc.ec.gc.ca/aqrb/index_e.cfm
4 Source: http://laws.justice.gc.ca/en/showdoc/cs/C-15.31///en?page=1.
5 1) Fuels Composition and Performance, 2) Novel and Advanced Internal Combustion Engines (ICE) Technologies, 3) Engine Hardware and Exhaust After-treatment, and 4) Health and Environmental Effects.
6 Two of the four T&I themes were relatively inactive. Although the Advanced, Efficient Powertrains and Fuels theme generated knowledge, it only conducted two projects. The other inactive theme, Intelligent & Efficient Transportation Systems, had one project.
7 Emission Factors (EF) can be used to estimate the rate at which a pollutant is released into the atmosphere (or captured) as a result of some process activity or unit throughput. (http://www.ec.gc.ca/pdb/ghg/guidance/calcu_fac_e.cfm)
8CCTII: Advanced End Use Efficiency: Transportation (TIB6): 2006-07 Annual Report.
9CO2 eq. is a unit of measure used to allow the addition of or the comparison between gases that have different global warming potentials. Source: http://www.ec.gc.ca/indicateurs-indicators/default.asp?lang=En&n=54C113A2-1#glossaryc, accessed on July 14, 2009.
10 Source: http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=38700, accessed on June 17, 2009.

RECOMMENDATIONS, MANAGEMENT RESPONSES AND ACTION PLANS

Recommendations Management Responses Responsible Official (Date)
1. The Energy Sector and the Innovation and Energy Technology Sector should develop an overall strategy to provide direction and priorities for the Transportation S&T Sub-sub Activity. The strategy should be supported by a governance mechanism that advances the delivery of the strategy by providing: coordination and synthesis of the programs; and, advice on strategic and operational issues such as targeting technology and policy receptors. NRCan’s Energy Enterprise (ES and IETS) accepts this recommendation.The Energy Enterprise has implemented a new over-arching governance structure for its RD&D, which includes grouping programs and projects from different funding initiatives into theme-based “Portfolios”. The Portfolios will be able to better manage and coordinate activities, and minimize the administrative burden associated with different funding initiatives. The Portfolios directly reflect and support strategic planning and reporting under the PAA Sub-Activity “Energy S&T”. The Clean Transportation Systems Portfolio will be initiating the development of a Transportation S&T Strategy that will build on the Portfolio’s existing Strategic Plan, and will set priorities, and guide the operational and funding decisions of NRCan’s Energy Enterprise regarding Transportation S&T. ADM/ES
ADM/IETS
March 2011
2. The Energy Sector and the Innovation and Energy Technology Sector should maximize the transfer of research, development and demonstration (RD&D) results to industry and government decision-makers, including a process for measuring and tracking the short and long-term uptake of RD&D results. NRCan’s Energy Enterprise accepts this recommendation. The Portfolio structure includes the participation of external advisors, representing diverse stakeholders, on program and/or Portfolio committees. To maximize the transfer of results, these stakeholders are engaged in both the planning and conduct of the RD&D. Results of RD&D activities are transferred to the broader transportation community, through activities such as workshops, program mid-year review meetings, the OERD Extranet, Energy Enterprise web sites, and presentations at key conferences. Additionally, projects with external industry and academic researchers, undertake technology and information transfer activities on their own. The project tracking template being developed by OERD (see response to recommendation 3), which will provide improved annual reporting at both the project and program level will allow for better tracking of the uptake of the Energy Enterprise’s RD&D results. NRCan’s Energy Enterprise acknowledges the need for incorporating knowledge management, including uptake where applicable, at the project level and is incorporating this as a mandatory requirement in program and project planning and annual project reporting. ADM/ES
ADM/IETS
Target implementation complete March 2011
3. OERD and CanmetENERGY (formerly CETC-Ottawa) should ensure that:
  • Financial information is complete and sufficiently detailed so that:
    • Programs and projects have planned funding and actual expenditure data readily available;
    • Programs and projects clearly identify funding and in-kind contributions from other sources;
    • Programs and projects avoid duplicate reporting (i.e., funding and results).
  • Programs maintain accurate lists of projects and electronic copies of project reports:
    • Projects which have been discontinued or cancelled are identified, including the reasons for which such decisions were taken;
    • In cases where project names change, cross-walks should be provided;
    • In cases where Web-based reporting approaches are used, publicly funded sites should remain publicly accessible.
NRCan’s Energy Enterprise accepts this recommendation. Given the broad level of participation in NRCan’s energy RD&D system, including public and private S&T performers, a project reporting and tracking template is being developed by OERD in consultation with the Strategic Evaluation Division (to be completed in 2009-10 and first used in f.y. 2010-11, for reporting on 2009-10). The template will enable monitoring, tracking and reporting activities at the project level, which in turn will facilitate and improve monitoring and tracking at program and portfolio levels. It will also meet the requirements for improved knowledge management (see response to recommendation 2).The tracking template will ensure that financial and project data are available on an interim basis, until the new department-wide financial and reporting system (Felix-SAP), is fully operational and is able to provide the data. ADM/ES
ADM/IETS
Project reporting template completed by
March 2010
4. Annual Reports should be based on the programs’ performance frameworks and report on performance indicators. The reports should provide context so that it is clear how the activity/project links to planned outcomes. The reports should include the interim steps made towards the uptake of the research and development results. NRCan’s Energy Enterprise accepts this recommendation.NRCan’s Energy Enterprise concurs with the requirement that Annual Reports address performance as defined by performance indicators in its planning documents. OERD is currently introducing a common project reporting template, developed in consultation with the Strategic Evaluation Branch (see Recommendation 5). OERD will be implementing a new annual report template which will draw on the project information provided and will summarize the information at the program level, and Portfolio level, clearly showing how the activities link to the planned outcomes. ADM/ES
ADM/IETS
The first set of Annual Reports will be available in the Spring of 2010 for program activities conducted in 2009-10.
5. Project Reports should be made less ambiguous by clearly defining tasks, milestones/timelines and resources so that the reports can be used to determine the degree to which projects are completing their planned tasks. The reports should provide context such that it is clear how the project links to planned outcomes. NRCan’s Energy Enterprise accepts this recommendation.In 2009/10, OERD will be implementing Project Status Reports, which have been developed in consultation with Strategic Evaluation Division. They will use a common template for all projects, will be clearly linked to the Portfolio structure, and will require, inter alia, clearly defined tasks, milestones/ timelines and resources, with clear links to Program logic models and performance frameworks. Additional context will also be provided. ADM/ES
ADM/IETS
The first set of project reports will be available in the Spring of 2010 for projects conducted in 2009-10.

1.0 INTRODUCTION

1.1 Overview of Report

This report summarizes the findings of an evaluation conducted on the Transportation Science and Technology Programs of Natural Resources Canada’s Energy Sector (the Clean Transportation Energy Sub-sub Activity 2.1.4.2 of the department’s 2008-09 Program Activity Architecture). The evaluation covers approximately $93.8M of NRCan funding for the period of 2002-03 to 2006-07. The programs within this Sub-sub activity are focused on achieving the result of finding new, long-term, cleaner and more efficient solutions to reducing environmental emissions by developing and disseminating new knowledge and new technologies through research, development and demonstration initiatives in transportation.

Six programs and elements of a seventh comprise this Sub-sub activity. The programs include approximately 335 projects:

  • The Advanced Fuels and Transportation Emissions Reduction (AFTER) Program is expected to lead to the development of new and innovative technology using new products and new processes that are marketable. Specifically, the AFTER’s contribution is expected to be manifested in new fuel and engine technologies designed to reduce emissions and produce a cleaner environment, on top of creating new markets and increasing hydrocarbon sales and oil sands crudes. It received $12.1M in NRCan funding from 2002-03 to 2006-07.
  • The Canadian Lightweight Material Research Initiative (CLiMRI) intends to develop and implement lightweight and high-strength materials in transportation applications for the purposes of reducing greenhouse gas emissions through improved vehicle efficiency through improving the competitive performance of the Canadian primary metals, automotive, truck, rail car and aircraft manufacturing industries and their associated parts suppliers. It received $7.1M in NRCan funding from 2002-03 to 2006-07.
  • The Particulate Matter (PM) Program aims to provide knowledge and tools that will support the development of technological and other measures to control and reduce emissions of particulate matter and its precursors from transportation sources. More specifically, the goal is to strengthen the scientific basis for policy and regulatory decisions affecting transportation-related emissions of particulate matter and its precursors. It received $5.8M in NRCan funding from 2002-03 to 2006-07.
  • The Technology and Innovation (T&I) Transportation Program’s objective is to advance the development and implementation of promising new technologies to achieve long-term mitigation of transportation’s contribution to climate change thereby strengthening Canada’s technology capacity for a more efficient transportation system. This objective is to be achieved through improvements to mainstream vehicle technologies such as internal combustion engines and transmissions. It received $6M in funding from NRCan from 2003-04 to 2006-07.
  • The Hydrogen Energy Economy (HEE) Program focuses on using hydrogen from renewable sources on applications such as automobiles and stationary power generators, fuel cells and other H2-powered. It received $31.4M in NRCan funding from 2002-03 to 2006-07.
  • The Canadian Transportation Fuel Cell Alliance (CTFCA) has two components, which are: to demonstrate the greenhouse gas reductions and evaluate different fuelling routs for fuel cell vehicles; and to develop the necessary supporting framework for the fuelling infrastructure, including technical standards, codes, training, certification and safety. It received $31.4M in NRCan funding from 2002-03 to 2006-07.
  • Some aspects of Technology Early Action Measures (TEAM).

Three of these programs (AFTER; CLiMRI; PM) are managed by NRCan’s Office of Energy Research and Development (OERD) as part of the Program of Energy Research and Development (PERD). The HEE Program is jointly delivered by PERD and NRCan’s CanmetENERGY Technology Centre-Ottawa (CETC-O). The Canadian Transportation Fuel Cell Alliance was managed by CanmetENERGY Technology Centre-Ottawa (CETC-O). The T&I Transportation Program was managed by OERD. Each of the programs evaluated functions as a separate entity although there are linkages between the CTFCA and HEE Programs.

The six programs are multi-sectoral and interdepartmental. Their activities encompass:

  • policy development including regulatory support,
  • research and development (R&D) activities ranging from basic to applied in nature,
  • development and demonstration of technologies, and
  • process development.

1.2 Transportation, Energy and Greenhouse Gas Emissions Program Description

Transportation plays a vital role in Canada—in 2006, transportation-related demand accounted for 12.2 percent of Canada's gross domestic product (GDP).11 The Canadian economy relies heavily on transportation for moving goods and services, making significant demands on the transportation sector. The importance of transportation to economic activity is highlighted by the following statistics:12

  • The Canadian transportation system carries over $1,000 billion worth of goods every year.
  • Nearly 16% of all personal spending is on transportation, and nearly 90 % of that is on personal motor vehicles.
  • Over the last decade, the Canadian transportation sector has experienced an annual average growth rate of 6.1%, almost doubling that of the economy at 3.3%.
  • In 2000, over 850,000 people occupied a job in the transportation industry or related functions, representing 7% of the Canadian workforce
  • Over the past 20 years, carriers’ costs have fallen in real terms by $10 billion, or 30%.
  • In 2000, over $20 billion a year was spent on the maintenance and operation of transportation vehicles and infrastructure operated by government and private operators.

Increases in the number of vehicles on the road and the size of these vehicles contribute to rising levels of greenhouse gas (GHG) emissions. Canada's per capita use of energy for road transportation (55.3 gigajoules per person) was higher than both the G7 (48.4 gigajoules per person) and Organization for Economic Cooperation and Development (OECD)-30 (38 gigajoules per person) averages for 200413, as illustrated in the following chart.

Figure 1: Transportation Energy Use per Capita, Road Transport, G7 Countries and the OECD-30 Average, 2004

Figure 1: Transportation Energy Use per Capita, Road Transport, G7 Countries and the OECD-30 Average, 2004

Figure 1: Source: Organization for Economic Cooperation and Development. OECD Environmental Data-Compendium 2006/2007. Transport Table 5 and General Data Table 1A. 2006-2007. Paris, France

The types of transportation vehicles addressed by the six programs evaluated are mostly on-road vehicles, i.e., passenger cars, light, medium and heavy duty trucks, and buses. These vehicles are of concern to manufacturers and policy makers because:
  • Transportation is the second largest source of GHGs in Canada (behind energy production), accounting for 27 percent of Canada’s emissions growth from 1990 to 2005. Over this timeframe, transportation carbon dioxide emissions increased by 33 percent (150 Mt to 200 Mt).14
  • Transportation sector GHG-related emissions includes over 70 kilograms (kg) of CO2 per gigajoule of energy, compared to the average 58 kg across all sectors, reflecting transportation’s reliance on traditional fossil fuels such as gasoline and diesel.15 This figure includes a 117 percent increase (24.3 Mt) in the emissions from light duty gasoline trucks, reflecting the growing popularity of sport utility vehicles.
  • Emissions from heavy duty diesel vehicles increased 19.4 Mt over the same period, indicative of greater heavy-truck transport. Offsetting these increases were reductions of 4.7 Mt from gasoline-fuelled cars and 1.4 Mt from alternatively fuelled cars.16
  • From1990 to 2005, GHG emissions from passenger vehicles increased by 10 percent, and passenger-kilometres increased by 30 percent during the same period.17
  • Between 1990 and 2004, the annual distance travelled by trucks increased by 43 percent. This led to a 71 percent increase in fuel consumption and a 73 percent increase in truck GHG emissions.18
  • Despite significant improvements in fuel efficiency, GHG emissions from on-road freight transport increased by 92 percent between 1990 and 2005 fro31Mt to 59 Mt. During this time, road freight activity, measured in tonnes-kilometres, increased by 160 percent, indicating improvements in fuel efficiency when compared with the increase in freight GHG emissions.19
  • While GHG emissions from the transportation sector continue to increase, emissions of related species, such as fine particulate matter, sulphur oxides, nitrogen oxides and volatile organic compounds, have shown a steady decline due to regulatory initiatives and stock turnover.

Carbon dioxide (CO2) is not the only GHG of concern to manufacturers and policy-makers. Other gases addressed by the six programs evaluated include components of smog: particulate matter, oxides of nitrogen (NOx), carbon monoxide (CO), and volatile organic compounds (VOC).

If current trends continue, GHG emissions from transportation will exceed 1990 levels by 32 percent in 2010 and 53 percent in 2020. The sources of emissions expected to grow most quickly between 1990 and 2020 are aviation (forecast to increase by 99 percent)20, off-road uses (diesel by 66 percent and gasoline by 57 percent) and on-road diesel (74 percent). On-road gasoline use is expected to increase by 44 percent between 1990 and 2020.

As the six programs evaluated demonstrate, addressing transportation environmental emissions (i.e., smog precursors such as particulate matter, ground level ozone and nitrous oxides and carbon monoxide and dioxide) involves industry (vehicle, component and materials producers; fuels and energy producers and distributors), government (federal, provincial, municipal), and the general public.

The approaches taken to reduce transportation-produced GHG emissions demonstrated by the six programs include:

  • Reducing vehicle weight while maintaining (or improving) performance, thus reducing energy consumption and emissions production (CLiMRI),
  • Developing and demonstrating technologies that produce no GHG emissions or substantially lower transportation GHG emissions (HEE, CTFCA), and
  • Investigating the composition of transportation fuels and how they react under Canadian operating conditions using newer vehicle technologies to support regulatory development (AFTER, PM).

11 Transport Canada. Transportation in Canada 2006: Annual Report. Ottawa, Transport Canada, 2007 (Cat. No. T1-10/2006E-PDF).
12 Transport Canada, Transportation in Canada 2002 – Annual Report, TP 13198E, ISBN 0-662-34028-0, 2003.
13 Text, chart and data table extracted from http://www4.hrsdc.gc.ca/indicator.jsp?lang=en&indicatorid=67
14 Transport Canada, Transportation in Canada an Overview (2007), page 10. Source: http://www.tc.gc.ca/pol/en/Report/anre2007/index.html Accessed May 28, 2009.
15 Transport Canada, Transportation in Canada An Overview (2007), page 10. Source: http://www.tc.gc.ca/policy/Report/anre2007/pdf/overview.pdf. Accessed May 28, 2009.
16http://www.ec.gc.ca/pdb/ghg/inventory_report/2007/som-sum_eng.cfm, accessed on May 27, 2009.
17 Transport Canada, Transportation in Canada 2007 – Annual Report, Cat. No. T1-10/2007E ISBN 978-0-662-48799-9 (2007).. Accessed at http://www.tc.gc.ca/policy/Report/anre2007/pdf/overview.pdf on May 28, 2009
18 Human Resources and Skills Development Canada, Indicators of Well-Being in Canada: Environment-Transportation (2009-05-08)
19 Transport Canada, Transportation in Canada 2007 – Annual Report, Cat. No. T1-10/2007E ISBN 978-0-662-48799-9 (2007).. Accessed at http://www.tc.gc.ca/policy/Report/anre2007/pdf/overview.pdf on May 28, 2009.
20 Options Paper of the Transportation Climate Change Table, Transportation and Climate Change: Options for Actions, November 1999.

2.0 TRANSPORTATION S&T SUB-SUB ACTIVITY

2.1 Overview

In order to address climate change issues, the Government of Canada has implemented a number of environmental/climate change initiatives since 2000, namely: the ecoEnergy Initiative, Action Plan 2000, Project Green, and the Climate Change Plan for Canada (including the Climate Change Technology and Innovation Initiative). As part of these initiatives, Natural Resources Canada implemented a number of transportation S&T programs designed to increase the development and rate of deployment of technologies that mitigate negative environmental impacts in the transportation sector. These programs comprise NRCan’s Program Activity Architecture Transportation S&T Sub-sub activity.

NRCan’s transportation energy science and technology investment is delivered through three major interdepartmental funding programs – Program of Energy Research and Development (PERD), Climate Change Technology and Innovation Initiative (T&I), and Technology Early Action Measures (TEAM) – as well as the Canadian Transportation Fuel Cell Alliance--a sunset program that was established by Climate Change Action Plan 2000 and ended in 2008. In May 2007, the ecoEnergy Technology Initiative was initiated.

The Transportation Energy S&T Sub-sub Activity 2.1.4.221 consists of programs directed towards research and development, and late-stage development and demonstration of technologies for promoting clean and efficient energy for the transportation sector. It encompasses transportation energy efficiency and optimisation, advanced fuels, policy and process development, and the characterization of the combustion and emissions reduction of those fuels, and hydrogen/fuel cells. These programs are intended to develop knowledge, technology, and processes that will be implemented by private sector partners and the policy and regulatory community.

NRCan program funders, along with other government funding agents and private sources of funding, foster the collaborative involvement of public and private S&T performers. The S&T performers include NRCan’s CANMET Energy Technology Centres, other public sector laboratories, academic and private sector laboratories/technical groups, as well as implementer and user communities and various standards bodies. These groups perform research, development and demonstration (RD&D) functions.

The programs that make up the Sub-sub Activity and that were evaluated are presented below in Figure 2:

Figure 2: NRCan Transportation S&T Program Structure

Figure 2: NRCan Transportation S&T Program Structure

It is important to note that Figure 2 presents the state of affairs during data collection for this evaluation and therefore excludes two transportation S&T programs that were terminated in 2003-04: Program at the Objective Level (POL22) 2.1.3 Transportation Fuel Systems and Program 2.2.4 Optimization of the Energy Efficiency of Transportation Systems. In addition, POL 2.2.2 Fuel Cells, Electric Hybrid vehicles, and POL 2.2.3 Hydrogen Initiative were merged to create POL 2.2.5 Hydrogen Energy Economy, which was then merged with the T&I Hydrogen Economy to create the Hydrogen Economy Program. The changes in transportation S&T programs through time are presented in Figure 3.

Figure 3: Evolution of Transportation S&T Programs

Figure 3: Evolution of Transportation S&T Programs

Annex A provides a summary of the Transportation S&T Programs, including program objectives and research themes.


21 The expected result of this Sub-sub activity is to find new, long-term, cleaner and more efficient solutions to reducing environmental emissions by developing and disseminating new knowledge and new technologies through research, development and demonstration initiatives in transportation.
22 Program at the Objective Level or POL implies ‘program’. During the course of this evaluation, OERD adopted the term “program’ to replace ‘POL’ in 2008.

2.2 Resources

Programs within the Transportation S&T Sub-sub Activity have different funding cycles with different start and end dates; therefore, the last five years of funding were used to show the total funding. Table 1 shows that NRCan provided approximately $93.8M (48%) of the total funding for the Transportation Sub-sub Activity from 2002-03 to 2006-07. Within the same period, other federal government departments contributed $29.1M (15%) and non-government sources, such as industry, academia, NGOs and international agencies, contributed $75.8M (38%).

Table 1:
Estimates of Funding Sources for NRCan’s Transportation S&T Sub-Sub Activity, 2002-03 to 2006-07 ($K)
Funding Sources 2002-03 2003-04 2004-05 2005-06 2006-07 Total % of Total
Natural Resources Canada 10,187 21,008 20,646 20,425 21,559 93,825 47
Non Government of Canada 4,410 18,334 19,129 15,522 18,413 75,808 38
Other Federal Government of Canada Departments 2,512 4,035 5,810 8,473 8,235 29,065 15
Grand Total 17,109 43,377 45,585 44,420 48,207 198,698 100

Figure 4: Estimates of Funding Sources for NRCan’s Transportation S&T Sub-Sub Activity, 2002-03 to 2006-07 ($K)

Figure 4: Estimates of Funding Sources for NRCans Transportation S&T Sub-Sub Activity, 2002-03 to 2006-07 ($K)

Table 2 shows the estimates of funding for the six transportation S&T programs from 2002-03 to 2006-07. NRCan funding is broken down by PERD, CCTII and NRCan A-base.23 Both PERD and CCTII provided approximately $42.1M and $42.07M, respectively. NRCan A-base amounted to $9.7M for the same period. Each program is identified with the agency responsible for leading the research and development and/or demonstration. The table also provides the percentage of funds received from various sources by each program, and the percentage of the total Transportation S&T funding distributed to each program. For example, Hydrogen Energy Economy and CTFCA received approximately $33M and $32M, respectively, of the $198.2M Transportation S&T funding.24

Table 2:
Estimates of Transportation S&T Programs’ Funding by Source, 2002-03 to 2006-07 ($K)

Program
Source 2002-03 2003-04 2004-05 2005-06 2006-07 Total % of Program Total % of Trans. S&T Total
Particulate Matter
POL 2.1.1
(EC)
OGD 1,005 1,170 1,240 3,680 3,133 10,228 56% 9%
PERD 759 679 713 1,429 1,265 4,845 27%
Non GoC 135 407 337 601 749 2,229 12%
CCTII       440 415 855 5%
NRCan A-base 0 0 0 50 40 90 0%
Sub-Total 1,899 2,256 2,290 6,200 5,602 18,247 100%
AFTER
POL 2.1.2
(NRC)
PERD 1,768 1,682 1,754 2,165 2,135 9,504 40% 12%
OGD 999 1,267 594 2,585 2,195 7,640 32%
Non GoC 638 576 70 1,801 1,242 4,327 18%
NRCan A-base 335 975 50 210 210 1,780 7%
CCTII       490 295 785 3%
Sub-Total 3,740 4,500 2,468 7,251 6,077 24,036 100%
CLiMRI
POL 2.2.1
(CANMET/ MTL)
Non GoC 1,537 517 1,257 1,154 1,154 5,619 42% 7%
PERD 807 999 993 995 995 4,789 36%
NRCan A-base 188 281 445 458 973 2,345 18%
OGD 308 15 23 110 110 566 4%
CCTII   60       60 0%
Sub-Total 2,840 1,872 2,718 2,717 3,232 13,379 100%
HEE
(includes POL 2.2.2, 2.2.3, 2.2.5, T&I)
(CETC/ HYFATE)
Non GoC 0 7,970 7,277 6,784 6,168 28,199 42% 34%
PERD 4,190 4,352 4,302 4,311 4,150 21,305 32%
OGD 0 1,581 3,155 1,199 1,367 7,302 11%
NRCan A-base 500 964 973 1,265 1,368 5,070 8%
CCTII 0 1,290 1,235 1,308 1,174 5,007 7%
Sub-Total 4,690 14,867 15,707 13,559 13,053 66,883 100%
T&I Transportation
(EC)
OGD 0 0 435 899 1,430 2,764 24% 6%
Non GoC 0 0 125 972 1,470 2,567 23%
CCTII 0 442 457 1,418 1,690 4,007 35%
PERD 0 0 199 1,040 420 1,659 15%
NRCan A-base 0 0 10 43 308 361 3%
Sub-Total 0 442 1,226 4,372 5,318 11,358 100%
CTFCA
(CETC/ HYFATE)
Non GoC 2,100 8,864 10,063 4,210 7,630 32,867 51% 33%
CCTII 1,640 9,284 9,515 4,803 6,121 31,363 48%
OGD 200 2 363 0 0 565 1%
PERD 0 0 0 0 0 0 0%
NRCan A-base 0 0 0 0 0 0 0%
Sub-Total 3,940 18,150 19,941 9,013 13,751 64,795 100%
Transportation
S&T Total
Non GoC 4,410 18,334 19,129 15,522 18,413 75,808 38% 100%
PERD 7,524 7,712 7,961 9,940 8,965 42,102 21%
CCTII 1,640 11,076 11,207 8,459 9,695 42,077 21%
OGD 2,512 4,035 5,810 8,473 8,235 29,065 15%
NRCan A-base 1,023 2,220 1,478 2,026 2,899 9,646 5%
Grand Total 17,109 43,377 45,585 44,420 48,207 198,698 100%

Sources of financial data for Tables 1 and 2 and Figure 4:

  1. OERD financial records contained the overall PERD and T&I expenditures and these records were used as the primary source of information for the PERD and T&I programs.
  2. Program Annual Reports contained budget information (i.e., not actual expenditures) by theme/activity area for PERD/T&I funding as well as for contributions (budgeted) from other funding sources. These financial records were used as OERD records do not contain this information.
  3. In most cases, resources provided by non-PERD/T&I sources did not distinguish between cash and in-kind.
  4. CTFCA was managed separately by NRCan’s CETC (CANMET Energy Technology Centre) and it tracked funding received from CCTII and CCAP. This financial information is approximate because the Program’s financial records were based on budget commitments and expenditures.
  5. In cases where sources of funding were categorized as “other”, they were reported as industry sources.

23NRCan A-base funding is on-going annual funding used for salary and operational maintenance.
24 The project funding and expenditures as reported by the individual programs differed from the overall funding per program reported by OERD. For the purpose of this evaluation, OERD financial figures for programs superseded all other financial record keeping.

3.0 EVALUATION SCOPE, METHODOLOGIES AND LIMITATIONS

3.1 Evaluation Issues and Methodology

This evaluation examined issues related to the six programs’ relevance, success and cost-effectiveness. The evaluation methodology was comprised of the following approaches:

Document Review – Over 200 documents were reviewed which encompassed each program’s documentation, plans and performance reports. In addition, Canadian federal S&T policy documents and priority identification instruments (e.g., the Budget Speech, Speech from the Throne and substantiating analyses, legislation, etc.) as well as related publications of other countries and federal departments, testimony to the House of Commons, reports to Parliament, and technical publications were reviewed.

Interviews – 61 interviews were completed. Interviewees included program managers, project leaders, industry stakeholders and partners. Samples typically were composed of members of the management and advisory committees involved in each program.

The 61 evaluation interviewees were selected from the following representative groups:

Evaluation Interviewees
NRCan OGD Industry University Provinces International Total
14 21 18 5 2 1 61

The breakdown by program is presented below:

AFTER: Thirteen interviews were completed. Six of the Program’s representatives declined to be interviewed. Out of the 13 that were interviewed, ten were federal government representatives and three were from industry.

Particulate Program: Seven interviews were completed. Four of the Program’s representatives declined to be interviewed. Out of the seven that were interviewed, all were federal government representatives. This sample is reflective of the Program’s reach which is mostly internal to the government.

CLiMRI: Fourteen interviews were completed. Three of the Program’s representatives declined to be interviewed. Of the fourteen that were interviewed, eight were federal government representatives, one was a university researcher, four were from industry, and one was from a foreign government.

T&I Transportation: Six interviews were completed. The evaluation team contacted 15 potential interviewees from a sample of 43 who were listed in the annual reports as members of the Expert Group and project leaders. There was a lack of familiarity with the Program among the interviewees, as a result, nine people declined to participate. Six federal government representatives were interviewed.

HEE: Eleven interviews were completed. Three of the Program’s representatives declined to be interviewed. Out of the eleven interviewees, four were federal government representatives, three were university researchers, and four were from industry.

CTFCA: Ten interviews were completed. Three interviewees declined, including the Program Manager. All of the ten interviewees were non-government representatives active on the Core Committee and at least one of the Program’s five working groups. Two interviewees represented non-government researchers; the remaining eight interviewees represented industry.

Case Studies – Twenty five in-depth case studies were completed. The case studies were conducted across a sample of projects to acquire more detailed knowledge of outputs and outcomes and are one of the major data sources for addressing the success of the programs. They involved a review of data and documentation as well as 61 interviews with project deliverers and stakeholders for each case study. The following table describes the breakdown of the interviews conducted for the case studies:

Case Study Interviewees
NRCan OGD Industry University Provinces International Total
18 16 24 1 0 2 61

Project Review - To supplement the document review and the case studies, a detailed review of 19 projects was undertaken. This involved an examination of project documentation to provide detailed information about the relevance and results of the selected projects.

3.2 Limitations

The Transportation S&T investment system is complex, involving many technical program areas and funding initiatives that are on different funding cycles. These programs involve a large number of federal departments, industry, universities, and other stakeholders. They cover a range of activities and objectives and exist as a loose cluster within the Transportation S&T theme of NRCan’s PAA. At the time of this evaluation, there was no planning or policy framework that encompassed the programs.

3.2.1 Delivery and Policy Structure

The complexity of the Transportation S&T investment system has been heightened by multiple policy statements over the years to guide the Federal energy and environment S&T actors. These policies include NRCan’s Energy Priority Framework and the 1999 S&T Companion Document, various Climate Change initiatives such as Action Plan 2000, federal Budget initiatives, etc. Consequently, during the time period covered by this evaluation, no single strategy outlined Federal objectives and priorities on Transportation S&T linking the programs together. There was also no logic model or results-based management and accountability framework (RMAF) linking the programs and their objectives together. (Since the end of the evaluation period, work has been undertaken to develop a strategy.)

As a result, this evaluation used the different policy statements on energy S&T and the existing RMAFs for individual programs as a starting point, integrated by application of a more comprehensive generic S&T results framework. This framework featured a broader value/ results model involving both private and public goods, proprietary and infra-technology benefits, and a balance of economic, environmental and other social science interests. Applying this framework allowed the evaluation to document the existing results of Transportation S&T in a more comprehensive manner.

A Context and Results Model was also designed (see the Results and Success section) that describes the types of impacts generated by the programs’ projects as well as key factors that ‘drove’ the projects reviewed. Four were identified: 1) Environmental concerns about GHG and black carbon; 2) Health concerns related to emissions; 3) Scarcity of non-renewable energy sources (fuel) and rising fuel prices; and 4) International agreements related to the environment and emissions.

3.2.2 Data Availability: Financial Data Issues

There were several issues with the Programs’ finances that limited an exact depiction of the expenditures on the S&T Transportation Sub-sub activity. The issues include: a) data inconsistencies; b) data labelling; c) actual expenditures versus budgets; d) double counting; and e) sub-contracting. (For details see section 4.3.2)

3.2.3 Data Availability: Performance Data Issues

The programs have not consistently monitored the results identified in their performance measurement frameworks. Instead, reporting was found to be largely focussed on achievement of technical project results and outputs, and usually does not include discussion of immediate and intermediate outcomes.

In some cases, the evaluation team experienced difficulties in obtaining the project reports themselves. Substantial effort was invested by the evaluation team in reconciling financial data, project titles and project codes from one year to the next. Project names varied from year to year and from the project financial databases to the annual reports. The reconciliation of project names and codes was essential to ensuring that ongoing projects could be tracked from one year to the next.

In some instances, it was difficult to determine whether a project was a PERD project or a T&I project. This is because T&I and PERD have projects with similar tasks, objectives, and titles. For example, when a program manager of a PERD Program at the Objective Level (POL) was asked to provide a list of the PERD POL projects, they included projects funded by T&I and included them as part of the PERD POL.

4.0 FINDINGS

4.1 Relevance/Rationale

This evaluation considered four questions in analysing the relevance of the six transportation science and technology programs:

  1. Do the programs continue to be consistent with federal and departmental priorities?
  2. Do the programs realistically address an actual need (i.e., to what extent do the investments respond to the needs and priorities of co-deliverers and targeted beneficiaries / clients)?
  3. Is it appropriate for NRCan to be responsible for the programs?
  4. To what extent would the research have been conducted in the absence of the programs?

Interviews, case studies, project reviews and document review were used to develop answers to the relevance questions. Over 200 documents were reviewed, including federal policy statements, departmental planning, policy and strategy reports, and science and technology policy related reports from Canada and other countries.

A wide range of program-specific documents were also reviewed (e.g., related to hydrogen) that were produced by the Government of Canada as well as multilateral organizations such as the Organization for Economic Cooperation and Development (International Energy Agency), or other countries such as the Government of the United States of America, and the United Nations Framework Convention on Climate Change. A host of NRCan publications and press releases were also reviewed.

In brief, all of the programs were found to be relevant to federal priorities, NRCan priorities, and the needs and priorities of stakeholders. One of the six programs, however, (the Particulate Matter Program) was found to have a closer relationship to Environment Canada’s mandate than that of NRCan. The priorities addressed by the six programs are related to the following federal and departmental priorities:

  • reducing emissions of pollutants, including greenhouse gases;
  • improving air quality and health by supporting technology and policy relevant research and development (R&D);
  • conserving natural resources;
  • supporting private sector competitiveness; and
  • increasing knowledge through S&T.
4.1.1 Were the Programs consistent with departmental and government-wide priorities?

Consistency with Federal Priorities
The ways in which these programs respond to federal priorities vary, but in all cases, the programs addressed federal priorities related to sustainable development/climate change as well as innovation and economic well-being. Some policy statements, such as the 2009 Budget, referred to some of the programs’ subject matter directly, such as hydrogen and fuel cell-powered transportation (i.e., the Hydrogen Energy Economy Program and the Canadian Transportation Fuel Cell Alliance).

The six programs were consistent with the federal priority of addressing sustainable development/climate change using the strategy of reducing exhaust emissions produced by transportation sources, as well as supporting Canadian private sector competitiveness. All of the programs were relevant to more than one federal priority. For example, in reducing transportation emissions to address climate change issues, new knowledge (e.g., properties of metals, alloys, and battery performance), processes (e.g., combining several vehicles components into one) and technologies (e.g., higher pressure containers for hydrogen, battery technologies) were developed that could increase the competitiveness of industry due to:

  • the value-added content of the technology involved;
  • the performance of the technology involved; or
  • as a result of the reduced operating costs achievable by implementing newly-developed processes.

Consistency with Departmental Priorities
The six programs were found to be consistent with NRCan strategic objectives; specifically those related to sustainable development/climate change and innovation, such as Strategic Objective 1: “Canadians derive sustainable social and economic benefits from the assessment, development and use of energy, forest and mineral resources, and have the knowledge to mitigate environmental impacts and respond effectively to natural and man-made hazards.”

The programs also were found to be relevant to NRCan priorities, such as the following two extracted from the 2008-09 NRCan Report on Plans and Priorities:

  • addressing climate change and air quality through science, technology and adaptation, and
  • advancing Canada's resource interests and sustainability efforts in the Americas and globally.

Some projects supported by a seventh program, the Technology Early Actions Measures (TEAM) Program, are included in the case studies portion of this evaluation. The projects supported by TEAM were also found to be relevant to supporting the preceding NRCan objectives.

The ways in which the programs supported these priorities include:

  • developing knowledge, technology and processes that improve the efficiency with which conventional energy (gasoline, diesel) is used by transportation, thus reducing exhaust emissions which contribute to climate change;
  • developing enabling policy instruments (e.g., installation and safety guidelines) that support and enable the use of new forms of energy (e.g., hydrogen) that have no or lower exhaust emissions;
  • demonstrating in real world conditions how villages, vehicles and refuelling stations using the new fuels operate; and
  • developing knowledge, technology and processes that (a) build on acknowledged areas of Canadian strength (e.g., fuel cell technology, materials expertise) and (b) add value by lowering the costs of the use of the knowledge and/or technology.

The Particulate Matter Program, while relevant to NRCan, is directly related to the priorities of Environment Canada as it is focussed on providing scientific data to support policy and regulatory decisions. The Program also remains appropriate to the mandate of NRCan, since they are aimed at increasing understanding of the environmental and health impacts resulting from energy use in the transportation sector. PERD is the part of NRCan whose mandate is most directly related to the PM Program since it is an interdepartmental research and development (R&D) program whose objective is to “to provide the continuing science and technology necessary for Canada to move towards a sustainable energy future.”

The objective of the Particulate Matter (PM) Program was originally defined in the 1999 Energy Science and Technology Companion Document as, “Support for the development of technological and other measures to control and reduce emissions of particulate matter.”25 The goal of this Program as presented in its first and subsequent Program Plans is: “To provide knowledge and tools that will support the development of technological and other measures to control and reduce emissions of particulate matter and its precursors from transportation sources.” The Program later qualified the original two objectives above by adding: “More specifically, the goal is to strengthen the scientific basis for policy and regulatory decisions effecting transportation-related emissions of particulate matter and its precursors.”26

The objectives articulated in NRCan’s Energy Science and Technology Companion Document and the ecoEnergy Technology Initiative are focussed on technology development. In comparison, the R&D activities supported by the PM Program are of a more policy focused, less-applied (i.e., fundamental research) nature. The types of tools and knowledge that are being developed by this Program, if successfully developed, transferred and implemented, are intended to play a role in providing sound science for policy decisions.27

As a result of changes to fuels and vehicles, more knowledge on transportation emissions is needed. The knowledge developed in this Program is intended to contribute to the Government of Canada’s regulatory agenda under the Canadian Environmental Protection Act (CEPA) 1999. Under CEPA 1999, the PM Program is directly aligned with the mandate and programs of Environment Canada, which has a legislated duty to conduct research related to particulate matter. Since 2000-01, Environment Canada has provided the overall coordination for the PM Program and has been the largest contributor of funds, contributing about 36 percent of the Program’s budget. The PM Program is also directly aligned with Health Canada’s legislated duty to conduct research relating to the role of substances, such as exhaust emissions, in illnesses.28


25NRCan Energy Sector, Energy Science and Technology Companion document (1999).
26 The PM Program was originally established to support the policy framework laid out in the 1999 Energy Science and Technology Companion Document, specifically objective 2.1.1 - support for the development of technology and other measures to control and reduce emissions of particulate matter.
27 The PM Program was originally established to support the policy framework laid out in the 1999 Energy Science and Technology Companion Document, specifically objective 2.1.1 - support for the development of technology and other measures to control and reduce emissions of particulate matter.
28 Source: http://laws.justice.gc.ca/en/showdoc/cs/C-15.31///en?page=1.

4.1.2 To what extent did the Programs realistically address an actual need?

Many changes are currently taking place in the transportation sector, in large part in response to new and planned regulatory changes addressing the interconnected areas of vehicle emissions, fuel composition, and energy efficiency. For example, in order to reduce air pollutant and greenhouse gas (GHG) emissions, the Government of Canada is pursuing a Clean Air Agenda to regulate criteria air contaminants (CACs) and GHGs. This includes increasingly stringent regulations under the Canadian Environmental Protection Act (CEPA) to reduce emissions of air pollutants from on-road and off-road vehicles and engines. Other legislation, such as the Motor Vehicle Fuel Consumption Standards Act, requires the development and application of new knowledge, technology, and processes in vehicle fuels, their emissions and vehicle technology. Performance of the fuels and technology under Canadian operating conditions and from Canadian fuel sources is a particular concern. For example low temperatures can precipitate wax from diesel fuels, and lower vapour pressures in gasoline. Relatively high concentrations of sulphur in fuel also have implications for refining.

The extent to which the six programs are related to these needs, expressed as infrastructure, policy, technology, and knowledge, were assessed through interviews, case studies, project reviews and the document review. The evaluation found that all of the six programs actively addressed identified needs and priorities of their private sector stakeholders and/or the needs of the sector as a whole.

Structurally, the governance of the programs—for example, decision-making with respect to the type of projects that the programs supported, the work priorities themselves and the design of the projects— were often performed with or by the programs’ partners through mechanisms such as advisory committees (e.g., CLiMRI, HEE) or work groups (e.g., Canadian Transportation Fuel Cell Alliance) in which stakeholders were actively involved and heavily represented. This approach ensured that a close relationship was established and maintained with industry priorities. However, in the case of some of the programs (e.g., CLiMRI and possibly T&I) it also had a negative effect in limiting what could be discussed among private sector competitors, and project proponents.

  1. Need for Infrastructure

    Infrastructure-related needs related to new fuels such as hydrogen are strongly influenced by a “chicken and egg” syndrome that forms a significant barrier to the introduction of these fuels and related technologies. This can be described as the interaction of three considerations:

    • reluctance of vehicle manufacturers to produce vehicles that operate on new fuels in the absence of a fuel delivery infrastructure to deliver these new fuels, combined with;
    • reluctance of consumers to purchase such vehicles given the lack of a fuel delivery infrastructure; and
    • reluctance of energy firms to construct a new fuel delivery infrastructure in the absence of sufficient demand, i.e., appropriate quantities of vehicles operating on the new fuels already in use.

    One of the six programs, the CTFCA, addressed the ‘chicken and egg’ syndrome. This was accomplished by conducting demonstrations of a range of vehicle, village and refuelling technologies using light (e.g., passenger cars), medium (e.g., buses) and heavy duty (e.g., trucks) vehicles. The demonstrations involved hydrogen refuelling stations (which it built), as well as fuel-cell powered vehicles.

    The HEE, addressed the infrastructure need by targeting research at gaps in technology and knowledge that have proven to be barriers to the development of appropriate infrastructure for fuel-cell powered vehicles. Examples include storage, reducing the cost of materials, improving performance, and development of transition technologies such as hybrid vehicles (powered by gasoline and fuel cells or diesel fuel and fuel cells).

  2. Need for Policy Instruments

    Introduction of new fuels and technologies can be delayed in cases where appropriate policy instruments are lacking, such as guidelines, regulations, and installation codes for basic safety and operability requirements (e.g., storage, installation, and delivery) of the new fuel and/or technology. Four of the six programs addressed the need for policy instruments for new fuels such as hydrogen (HEE together with the CTFCA) and for existing technologies (AFTER and the Particulate Matter Program).

  3. Need for Technology

    Policies such as the Clean Air Agenda, CEPA, and proposed legislation and supporting regulations such as the Renewable Fuels Content Regulations under CEPA specify minimum content of renewable fuels in gasoline, and limit allowable production of components of GHGs and criteria air contaminants (CACs) in exhaust. The Motor Vehicle Safety Standards contain requirements for the vehicle crashworthiness, which directly affects the use of light weight new materials to improve new motor vehicle fuel efficiency (improvements in automotive fuel efficiency are being pursued by Canadian and American policy). Strong understanding of the technological, environmental and human health impacts of transportation technologies (which are rapidly developing) under Canadian conditions is important to support technology development, as well as to inform policy and regulatory decisions.

    As noted previously, compliance with these policies was found to have created a strong need on the part of government as well as industry for improvements in current transportation technologies. This evaluation found that the programs addressed the need for technology as follows (these are examples only):

    • AFTER: particulate matter filters and sensor technologies;
    • CLiMRI: advanced forming and fabricating technologies;
    • HEE: fuel cell, fuel cell engine and storage technologies;
    • CTFCA: evaluating the performance of hydrogen technologies;
    • PM: tools and methods to analyze PM; and
    • Technology and Innovation (T&I) Transportation Program: supporting CLiMRI and AFTER.

    Related to the technology need was an industry and academia need for ready access to laboratory facilities, knowledge, and expertise to address technological and process-related challenges in areas such as light weight materials. All of the six programs were found to be responsive to this need in that they made NRCan laboratory facilities, knowledge, and/or expertise, not available elsewhere, accessible to program stakeholders.

  4. Need for Knowledge

    The six programs advanced the state of knowledge in their specific areas of focus. This knowledge was related to policy needs as well as to technology development needs. In the policy area, the knowledge advanced understanding of the causes and effects of areas being regulated (e.g., PM, AFTER, HEE) and was used to support the development of models, which in turn, serve as inputs into policy development (AFTER, PM).

    Knowledge was also developed by the programs that provided input to development of technology and processes (e.g., CLiMRI, HEE, and CTFCA).

    Specifics types of knowledge generated by the six programs are:

    • AFTER: impact of fuel composition on exhaust emission, impact of technology on fuel efficiency and exhaust emissions under Canadian conditions as input to regulatory and technology development;
    • CLiMRI: new light weight materials, supporting manufacturing applications and process development;
    • HEE: hydrogen technology supporting development of policies, codes, standards, and regulations;
    • CTFCA: performance of hydrogen fuelling stations and hydrogen-fuelled vehicles;
    • PM: exhaust emissions constituents supporting regulatory development; and
    • Technology and Innovation (T&I) Transportation Program: supporting CLiMRI, PM, and AFTER.
4.1.3 Was it appropriate for NRCan to be responsible for the Programs?

When assessing the appropriateness of NRCan’s leadership role in the six programs, consideration was given to the NRCan mandate and legislation as well as the mandates of NRCan’s partners.

Activities carried out by the six programs are relevant to NRCan’s mandate as described in the Department of Natural Resources Act,29 the Resources and Technical Surveys Act,30 and the Energy Efficiency Act.31

The evaluation found that all of the six programs encompassed by this evaluation are related to NRCan’s mandate in that they:

  • are concerned with sustainable development and/or the wise use of Canada’s resources (both energy and mineral);
  • involve natural resource-related R&D, chiefly energy;
  • develop capacity among the stakeholders and transfer technology through R&D partnerships;
  • develop policy related to natural resource (energy) use;
  • provide knowledge needed to support policy development; and
  • are oriented towards improving the competitiveness of Canadian industry.

NRCan’s role in the programs was found to be appropriate for all of the programs. The appropriateness of NRCan’s role was attributed to NRCan’s mandate areas as prescribed in legislation as well as to the department’s expertise with respect to:

  • industry (energy producers, vehicle producers, component manufacturers, minerals and materials producers);
  • the field of energy as a whole (conventional and alternative), and materials; and
  • NRCan’s technological expertise, which is recognized internationally, in several fields such as metals technology, fuel cells and in some areas such as the development of new catalysts that can use the reductant possibilities of an hydrogen-carbon monoxide mixture which is unique in the world.

Although other federal departments have working knowledge in some of these areas, none has expertise in all of them; nor are they recognized as being field leaders as NRCan is.

The sixth program evaluated, the Particulate Matter (PM) Program, is consistent with government wide priorities on reducing environmental and health impacts from emissions. It is unique, however, as in comparison to the objectives articulated in NRCan’s Energy Science and Technology Companion Document and the ecoEnergy Technology Initiative which are focussed on technology development, the R&D activities supported by the PM Program are of a more policy focused, less-applied (i.e., fundamental research) nature.

The PM Program remains appropriate to NRCan’s PERD, since PERD is an interdepartmental R&D program whose objective is “to provide the continuing science and technology necessary for Canada to move towards a sustainable energy future”. The Program aligns directly with the mandate and programs of Environment Canada, which has a legislated duty under CEPA 1999 to conduct research related to particulate matter. The PM Program is also directly aligned with Health Canada’s legislated duty to conduct research relating to the role of substances, such as exhaust emissions, in illnesses.32

NRCan’s authority to conduct research is established under the Energy Efficiency Act, 1992 under Part II section 21, which states that the Minister may, for the purpose of promoting the efficient use of energy and the use of alternative energy sources, (a) conduct, or cooperate with persons conducting, research, development, tests, demonstrations and studies; (b) publish information, research or test results; (c) assist, cooperate with, consult and enter into agreements with any person, including any department or agency of the Government of Canada or of any province; (d) make grants and contributions; and (e) undertake such other projects, programs and activities as in the Minister’s opinion advance that purpose.

Thus, the PM Program can be linked to NRCan activities, and directly linked to Environment Canada, which has a legislated duty, under section 44 and under section 158 of CEPA 199933, to:

  1. conduct research and studies relating to pollution prevention, the nature, transportation, dispersion, effects, control and abatement of pollution and the effects of pollution on environmental quality, and provide advisory and technical services and information related to that research and those studies;
  2. conduct research and studies relating to (i) environmental contamination arising from disturbances of ecosystems by human activity…..and section 158 of CEPA 199934, to:
  3. undertake research and development programs for the study of the effect of vehicles, engines or equipment or emissions on air pollution, energy conservation and the environment and for the promotion of measures to control that effect;”

In order to implement its mandate, Environment Canada’s Air Quality Research Branch (AQRB) conducts research on the transport, transformation and deposition of air pollutants and monitors atmospheric constituents such as stratospheric ozone, acid rain, photochemical smog; greenhouse gases, hazardous air pollutants and fine particulate matter. The specific goal of particulate matter research in the AQRB is to develop the tools and knowledge to evaluate possible fuel and other transportation-related standards (e.g., emissions, fuel efficiency), which may be needed to meet the future particulate matter air quality standards.

As previously noted, Health Canada is also a participant in the PM Program and also has a legislated duty, under section 45 of CEPA 1999, to (a) conduct research and studies relating to the role of substances in illnesses or in health problems; (b) collect, process, correlate and publish on a periodic basis data from any research or studies done under (a); and (c) distribute available information to inform the public about the effects of substances on human health.

In terms of participation in the Program, Environment Canada received 55 percent ($3.9M) of the funding over the lifetime of the Program. NRC received 22 percent ($1.5M), HC 16 percent ($1.1M), TC two percent ($141K), Royal Military College of Canada (RMCC) three percent ($202K), and NRCan three percent ($193K). Environment Canada was the largest contributor to the Program, providing 36 percent ($7.6M) of the total funding from 2001-02 to 2006-07. (See Annex C & D for more details).

Despite the strong role that EC is playing in the Program, interviewees reported that it is appropriate for NRCan to be responsible for the Program because NRCan’s mandate relates to energy and the use of energy, while EC focuses more on environmental protection. Interviewees commented that they view NRCan as the leader in horizontal R&D and that without the involvement of PERD a silo effect might develop and the research would lose its links with health effects.


29 Source: http://laws.justice.gc.ca/en/showdoc/cs/N-20.8/bo-ga:s_5//en#anchorbo-ga:s_5 accessed on May 15, 2009.
30 Source: http://laws-lois.justice.gc.ca/eng/acts/R-7/index.html accessed on May 15, 2009.
31 Source: http://laws.justice.gc.ca/en/ShowFullDoc/cs/E-6.4///en accessed May 15, 2009.
32Source: http://laws.justice.gc.ca/en/showdoc/cs/C-15.31///en?page=1.
33Source: http://laws.justice.gc.ca/en/showdoc/cs/C-15.31///en?page=1.
34Source: http://laws.justice.gc.ca/en/showdoc/cs/C-15.31///en?page=1.

4.1.4 Incrementality (What would likely have happened had the programs not existed?)

Given the high priority allocated by the Government of Canada as well as by governments of other countries to addressing climate change issues, and because all six programs evaluated are aimed at improving energy efficiency and reducing GHG emissions, the research that they support would likely have been carried out elsewhere had these programs not existed. However, interviewees indicated that the work would likely have been carried out at a slower pace, with reduced scope, and without the credibility provided by the involvement of the federal government/NRCan. The lack of federal/NRCan participation would have made attracting funding more difficult.

The AFTER Program is an exception to this. The case study and interview data indicate that the incrementality of the Program is project-specific. What would likely have happened had the AFTER Program not existed varies from project to project. The case study for the Advanced Catalytic Materials and Concepts for Diesel Exhaust After-treatment project showed that there are many players involved in the search for a way to reduce the amount of oxides of nitrogen contained in diesel engine exhaust. However, no other research teams are using the approach that the NRCan-supported team is using. Thus the line of R&D inquiry would likely have continued, but whether the technology approach itself would have been pursued is not known.

In the case of another project (Advanced Engines, Emission Controls and Fuels Performance) whose objective was to assess the effects of advanced low sulphur diesel fuels diesel engines and emission control systems, the work (according to industry) would have been pursued in the absence of PERD funding. The impact would have been that the work would have had a reduced scope, slower pace, and without the benefit of the expertise of EC’s Emissions Research and Measurement Division, which was reported as being quite valuable.

Overall, at least some of the transportation S&T work would have moved to other countries (e.g., hydrogen/fuel cell development and demonstration), thus resulting in a loss of Canadian expertise, income, technological leadership and possibly resulting in a delay or loss of technology that responds to Canadian operating conditions (e.g., cold weather has significant impacts on the operation of fuel cells).

More specific comments regarding the incrementality of the remaining five programs are:

  • CLiMRI: Interview data indicates that there are private sector laboratories which could in theory carry out CLiMRI type projects had the Program not existed. However, these laboratories do not necessarily have the competencies required and would likely attempt to purchase it. It was also reported that metallurgy competencies are very difficult to find. All interviewees consistently noted that the expertise and facilities of MTL are unique and valuable and that CLiMRI “helps build the synergy between industry-government and academia.”

    Other government departments such as Transport Canada or the National Research Council are potential alternative delivery choices, however, they do not have the research facilities, materials expertise and networks that MTL does. NSERC (the Natural Sciences and Engineering Research Council) and universities were described as being unable to carry out this type of R&D because of the lack of facilities and the dangerous nature of magnesium, which requires special equipment and training to handle.

    The newly-established (2008) Automotive Innovation Fund (AIF) also funds similar research to CLiMRI on a larger scale. However, it is unlikely that the AIF could deliver the results achieved by CLiMRI to date had CLiMRI not existed. This is due to the value attributed by interviewees to the involvement of NRCan’s Materials Technology Laboratory (MTL), which supports CLiMRI.

  • CTFCA: Had the CTFCA Program not existed, linkages among government, industry and other stakeholders would not have occurred; some projects would have continued but at a slower pace or would not have been undertaken at all; and projects with large public good benefits would likely not have been undertaken or would have been delayed (e.g., bus demonstrations). Alternative sources of funding would have been required, but would have been more difficult to obtain due to the lack of the government’s presence, which was described as having lent credibility, thus helping to attract investments.

    In addition, had the CTFCA Program not existed, some development work would likely have moved to countries where government funding was available, resulting in a loss of Canadian expertise. Less foreign involvement would likely have occurred and the Canadian hydrogen infrastructure, including regulations, would have varied from province to province with no national system. Buy-in from industry and end-users would probably also have been lower.

  • HEE: Without NRCan or federal government involvement, hydrogen R&D is of sufficient importance that some of it would probably have continued, but at a slower pace and reduced scope. This means that the likelihood of results appropriate to Canadian operating conditions being achieved would also likely have been lower. The amount of collaboration would have decreased, given the role NRCan plays in bringing stakeholders together and the likelihood of results taking place would also have diminished.
  • PM: Had the PM Program not existed, some of the research would likely have been carried out by each of the five partner federal departments separately, with each department focussing more on its own interests (opposed to working collaboratively). The research would also be reduced in scope as particulate matter is only a small priority for any given department.
  • T&I: This issue was not explored since the Program ended in 2007-08.

4.2 Results and Success

4.2.1 To what extent have the Programs achieved or made progress towards intended outcomes?

To illustrate the types of results that the Sub-sub Activity has contributed to, a Context and Results model is presented below in Figure 5. The model portrays the types of results throughout the impact chain, as well as contextual factors that surround the Programs and their projects. As determined by the evaluation, various contextual factors determine the extent to which the Programs were able to be successful. Contextual factors identified as key drivers of these Programs include: 1) environmental concerns about GHGs and black carbon;35 2) health concerns related to emissions; 3) scarcity of non-renewable energy sources (fuel) and rising fuel prices; and 4) international agreements related to the environment and emissions.

The second column of boxes represents the Programs that lead to the results. Also indicated is the contribution of the project partners (including other research organizations and private sector firms). This participation is considerable in terms of cash and in-kind contributions.

The next set of columns in the model illustrates the impact chain, going from research outputs to actual impacts for Canadian society. As shown, the Programs generally lead to better information, as well as improved fuels, materials, and technologies. At this point, the impact chain is more and more influenced by external factors and players, although the evaluation evidence shows that the Programs are key determinants of some of these impacts, such as new manufactured products, infrastructure and better informed policy-makers and regulators. The last two columns illustrate some of the impacts to which the Programs contribute. These include better air quality, safer equipment, reduced fuel consumption, as well as economic impacts including revenues for Canadian firms, and savings for consumers and transportation companies.

In terms of attribution, the Programs are not the sole contributors to the impacts. However, some of the projects would not have gone ahead without NRCan’s contribution, while others would have gone ahead but with reduced scopes. This indicates that the NRCan funding played a substantive role in the achievement of these outcomes.

Figure 5: Context and Results Model

Figure 5: Context and Results Model

Summary of Economic and Environmental Impacts
Apart from the number of publications and patents, at least three types of actual and potential impacts can be compiled from the Programs: GHG reductions, economic activity generated, and cost savings generated. As shown below, the following projects have directly contributed, or are likely to lead to, the described impacts. For further details please see the section below on the results of each Program as well as the Cost-Effectiveness section:

GHG reductions:

  • CLiMRI: Lightweight Vehicle Body Architecture Project: 6,000 metric tonnes by 2014 (if implemented in 2012).
  • Transportation T&I: Reduction in Fuel Consumption via Reduction of Aerodynamic Drag: As of 2008, 110 truck trailers have adopted these components, which will – in the most pessimistic scenario – result in 800,000 litres of fuel and 2,160 metric tonnes carbon dioxide equivalent (CO2 eq.)36 in savings over five years. If 8,000 truck trailers (five percent of Canadian fleet) adopted the components the likely reductions would be 157,000 metric tonnes over five-years.
  • CTFCA: The Vancouver Fuel Cell Vehicle Program: 18,500 metric tonnes in actual savings. (Timeframe: 2005 to 2008, with 168,000km travelled.)
  • TEAM: The TEAM program developed the System for Measurement & Reporting of Technologies (SMART) process to determine the GHG emission reductions achieved in demonstration projects and applied this methodology to the transportation demonstration projects funded during the period encompassed by this evaluation. TEAM developed detailed GHG emission reduction forecasts for each project funded that could be used as reference documents for future transportation demonstration projects. The Program also developed a sector-specific protocol called "Fuel Cell Transportation Sector Specific protocol" which can be generally applied to such projects. The TEAM SMART process was submitted to a United Nations ISO standards committee and is the basis of a new, international standard (#14064 part 2),37 which now applies to any GHG emission reducing project.

Total: 20,660 metric tonnes of CO2 eq in actual savings over a five-year period.

Cost savings to Canadians resulting from better efficiencies (reduced fuel consumption):

  • CLiMRI: Lightweight Vehicle Body Architecture Project: $2.6M (if implemented).
  • Transportation T&I: Reduction in Fuel Consumption via Reduction of Aerodynamic Drag: To date, 110 truck trailers have adopted these components, which will – in the most pessimistic scenario – result in $0.8M in savings over five years. If 8,000 truck trailers (five percent of Canadian fleet) adopted the components the likely cost savings would be $58M over a five-year period.

Total: $0.8M in actual cost savings over a five-year period.

Revenues and sales generated:

  • CLiMRI: Galvanic Corrosion: $400K in sales for two companies, from 2005-2007.
  • CTFCA: The Hydrogen Highway Project: $3.5M in sales (estimated sales in 2008 and 2009 could total $15M).
  • T&I - TEAM: Home natural gas refuelling appliance: $6M in annual sales, from 2005 to 2008.

Total: $9.9M in actual sales, from 2005 to 2008

The following section summarizes the results and outcomes of each of the programs.

AFTER
Finding: The Program has made progress towards achieving its immediate outcomes in that it has generated and published findings related to its four research themes and has developed prototypes related to particulate matter filters and sensors. However, at this time, there is limited evidence that immediate outcomes are contributing to intermediate and final outcomes.

The majority of the Program’s outcomes are related to advancing understanding. The research has resulted in more than forty-five journal publications, and produced forty-four conference and technical meeting presentations. ‘Advancing understanding’ is an outcome that requires definition of a results-based end point (e.g., advancing understanding to the point that the impacts of proposed regulations X, Y and Z on engine performance have been defined). Such definition is needed in order for program stakeholders to know at what point work in a given area will be considered sufficiently complete. Such definitions do not currently exist. In addition, achievement of some outcomes such as reduction of GHG emissions requires technology development, transfer and implementation. The latter two processes are the purview of the AFTER Program’s industry partners and are not under the control of the Program itself.

With respect to technology development, the Program has developed three pre-commercial sensor prototypes that monitor engine performance to control emissions that are currently being tested by industry. A patent application has been filed for one of these technologies and another has had its patent application approved.

Examples of areas where the Program has generated knowledge are:

  • The composition and properties of Canadian ultra low sulphur diesel (ULSD) fuel derived from both conventional and oil sands and biodiesel blends and their effects on soot formation and emissions - The preliminary results suggest that fuels with higher hydrogen contents tend to produce somewhat lower particulate matter and NOx emissions.
  • The parameters affecting HCCI combustion, fuel enrichment and reforming, and associated emissions - The results may lead to new approaches to engine design that reduce the tendency to rely on complex emissions controls which reduce fuel efficiency and overall engine performance.
  • New/novel catalytic solutions to remove NOx from diesel exhaust - Preliminary testing of the catalyst has found an increase in the efficiency of engines of 5 to 10 percent.
  • The relative mutagenicity and toxicity of biodiesel fuels, low sulphur diesel, ultra-low sulphur diesel fuels and various blends and their respective combustion products; and on the environmental toxicity of biodiesel, particularly to aquatic organisms. Preliminary research indicates that the toxicological hazard of emissions from engines operated using biodiesel-derived fuel or biodiesel blends are similar to or greater than those from using reference fuels; and that biodiesel fuels are less toxic in the environment than petro-diesel (for example on aquatic test species), although this may be offset by biodiesel fuels’ ability to persist longer in the environment.

Examples of pre-commercial technologies to control and reduce emissions that were developed include:

  • A particulate matter (PM) sensor for monitoring PM from diesel engines has been developed and is being tested by industry. A US patent application has been approved. The sensor allows for the detection of excessive PM in exhaust so that adjustments can be made to improve engine efficiency which subsequently reduces the production of emissions.
  • An Exhaust Cyclic Variability (ECV) combustion stability sensor has been developed. This sensor allows for the detection of engine combustion stability problems so that adjustments can be made to stabilize combustion before the problems increase exhaust emissions. A US patent application has been filed. Prototypes have been developed and tested using lean-burn, turbocharged diesel and natural gas engines.
  • An active particulate filter regeneration system was developed for vehicles that operate under conditions that do not allow passive regeneration of the filter. This system has been validated using ULSD and biodiesel blends.

Overall, interviewees felt that the Program was progressing towards achieving its objective and felt that its work on fuels, engine technologies and health impacts was contributing to the field of knowledge in these areas. Several interviewees emphasized the importance of the Program’s work in comparing oil sands derived fuels to conventional sources, and an oil industry representative commented that the fuel chemistry work carried out through this Program is adding to the knowledge base in this area. Interviewees reported that as a result of AFTER’s work they have a better understanding of how fuels interact with engines and of the health risks involved.

In terms of uptake of results outside of the Program, there is limited evidence that the work on fuels has helped industry optimize fuels yet, however, as referenced above, the PM prototype sensor and two other prototypes are being tested by car manufacturers.

Particulate Program

Findings:

  • The Program has generated knowledge related to the role of transportation in the production of particulate matter, the evolution of particulate matter in the atmosphere and the human health impacts of particulate matter. The Program’s outputs are mainly, but not exclusively in the form of published papers, presentations at conferences, and model development and testing.
  • The Program’s research results could in the near future influence the policy and regulatory communities. However, at this point in time, there is limited evidence that the knowledge has been used to strengthen policy and regulatory decisions.

The Program has generated a number of outputs that fall into the categories of: emission characterization; PM measurement tools; PM chemistry; databases; PM modeling; PM health effects. The outputs demonstrate the relevance and quality of the projects supported by the Program, and provides the knowledge for scientific exchange, collaboration and information dissemination. According to interviewees, the quality and quantity of data has significantly improved over the Program’s life span.

The Program has generated a number of outputs that fall into the categories of: emission characterization; PM measurement tools; PM chemistry; databases; PM modeling; PM health effects. The outputs demonstrate the relevance and quality of the projects supported by the Program, and provides the knowledge for scientific exchange, collaboration and information dissemination. According to interviewees, the quality and quantity of data has significantly improved over the Program’s life span.

The Program has made incremental progress by improving the knowledge base related to particulate matter and its precursors from ethanol blends, advanced technology vehicles, and railway locomotives. It developed and enhanced tools and methods to analyze PM and to produce information that could inform upcoming policy and regulatory development processes. The Program has disseminated its research results including 25 refereed/peer reviewed publications, and 11 national/international presentations. Two of the publications have received several awards.

The Program’s main success may be its horizontal approach to its subject matter, which examines the whole PK spectrum from tailpipe emissions to cardio-respiratory impacts, thus generating knowledge and awareness on potential health effects area and ensuring that they are not under-emphasized.

The key clients for this research are the policy and regulatory communities in the federal government and other jurisdictions. Despite the noted scientific achievements, interviewees had varying views on whether the science generated in this Program has influenced policy to date.

Several interviewees felt that the research had contributed to policy in some way, but could not provide any examples. Two interviewees provided the following three examples:

  • The city of Hamilton has used the Program’s data for decisions about banning idling and changing bike routes to avoid high pollution areas.
  • The Program provided data on the differences among diesel fuels to the city of Ottawa to inform vehicle fleet decisions.
  • The Program published its research findings on health risks associated with fine and course particles. Since Health Canada uses published material to evaluate the health risks of fine and course particles in its risk assessment work, it can be assumed that work published by the Program would be reviewed during this process. The risk assessment documents are used in developing Canada-wide standards with national stakeholders as well as regulations under CEPA 1999. Through this chain of events, Program research can be expected to serve as input into a range of policy and regulatory processes.

In a general sense, it was felt that information from the PM Program and other programs contributed to the creation of the Canada Wide Standards, Ozone Standard, PM Annex and was used in negotiations with the U.S. on particulate matter from transportation. All interviewees agreed that the Program has made significant advances in modeling and that the research could in the near future influence decisions. One of these interviewees felt that it is not realistic to judge the Program on its contributions to policy because the current Program was designed primarily to generate knowledge and is one small part of a global effort to study particulate matter.

All interviewees noted the apparent disconnect between the policy groups and research and development. To date, the efforts that the Program has been able to exert in communicating and influencing the policy community relies heavily on the initiative of key individuals in the Program, rather than on mechanisms within the Program structure and interdepartmental governance.

CLiMRI
Findings:

  • The Program generated knowledge of lightweight materials that can be used to make efficient designs and components.
  • The Program developed knowledge (e.g., processes, technology pathways) that can contribute to decreased operating costs of using lightweight materials.
  • CLiMRI’s expertise is well-recognised in the area of designing and processing metals technologies.

CLiMRI made advances in the areas of aluminum, magnesium, ultra-high strength steel (UHSS), titanium, process development and reduction of manufacturing costs. Some of these accomplishments are discussed below. From 2003-04 to 2006-07, CLiMRI scientists made 40 presentations at national and international conferences and published over 80 technical papers.

  1. Aluminum
    CLiMRI research findings relate primarily to process development. They included finding ways to reduce the costs of aluminum alloys used to produce gasoline engine blocks and other automotive power-train components, and to improve their material performance and production costs. The research made significant advances in the area of aluminum hydroforming by addressing manufacturing challenges. Tube hydroforming is important because of its potential to significantly reduce the cost penalty of using aluminum for fabrication of structural components to achieve weight savings.
     
  2. Ultra High Strength Steel
    CLiMRI helped reduce some manufacturing challenges related to forming and fabricating technologies for UHSS tube-hydroformed components. This work enabled MTL-CLiMRI to become a leader in welding advanced UHSS tubes. Because this R&D is relatively early in the application of UHSS to hydroforming automobile parts, benefits that have been identified are anticipatory. A 10 -15 percent weight savings has been achieved to date from using hydroforming to replace stamped parts.
     
  3. Pilot Scale Tube Facility
    Another significant result was the development of a pilot scale tube making facility at MTL. The pilot facility allows testing to take place that is very similar to actual industry production. This is important because it reduces development time and R&D costs that industry would otherwise have incurred.
     
  4. Magnesium
    The majority of advances were made by CLiMRI in the area of magnesium. One key example is the “Forming of Magnesium Sheet for Vehicle Weight Reduction” project, which developed advanced material knowledge and options for the use of lightweight materials in the design and manufacture of lower-weight components.38 An interviewee reported that the knowledge and publications generated by this project are believed to have been used in the production of a truck and lift-gate of a specific upscale car model.  The project also generated interest from a variety of sources including a major automotive manufacturer, a component manufacturer, an Austrian company and the U.S. Department of Energy (DOE) with respect to the Structural Cast Magnesium Development (SCMD) project.39

    A magnesium engine cradle was developed by a project involving CLiMRI, the U.S. DOE, the U.S. Automotive Materials Partnership, and corporations from the U.S. and Canada. MTL-CLiMRI was instrumental in preparing a draft paper on bolt-load retention standards for the Society of Automotive Engineers International (SAE) and the United States Council for Automotive Research (USCAR). With CLiMRI’s work on Galvanic Corrosion and Bolt Load Retention for magnesium in the SCMD project, MTL-CLiMRI gained international recognition and encouraged Canadian industry and universities to focus on lightweight materials research by increasing knowledge and awareness.40

    Evaluation case study data indicates that the two outputs, galvanic corrosion control strategies and bolt load retention testing procedures were incorporated into the commercial design and production of the 2005-07 magnesium engine cradle of a domestically produced sports car. The projects’ contributed to approximately $400K in sales for two companies. As a main impact, this project demonstrated that lightweight materials such as magnesium can be used in cars to reduce weight.41

    The success of the magnesium engine cradle contributed to the formation of an international (Canada-China-U.S.A.) collaborative project for developing a magnesium-based front-end structure to further improve fuel savings (see below for the Multi-Material Lightweight Vehicle Body Architecture (mmLiVBA) Project). The work conducted by CLiMRI on galvanic corrosion resulted in an NRCan patent application and a Canadian auto parts company has expressed interest in the process for use in their proprietary hybrid (steel-magnesium) components.42

  5. Multi-Material Lightweight Vehicle Body Architecture (mmLiVBA) Project
    The most illustrative example of CLiMRI-supported research is the multinational Multi-Material Lightweight Vehicle Body Architecture Project (mmLiVBA). The mmLiVBA is a series of smaller projects aimed at a common objective of achieving “substantial weight savings (approximately 40 percent) in vehicles through the combined use of advanced high-strength steels in a re-designed safety cage, coupled with the use of very light magnesium alloys for the front-end structure.”43

    The mmLiVBA is being carried out by an international consortium in which Canada participates through CLiMRI. CLiMRI’s role in the mmLiVBA is to develop forming and welding techniques, and to manage mmLiVBA projects that the Canadian government funds. The United States Automotive Materials Partnership (USAMP) provides a coordinating role internationally.44

Economic Impacts
If Phase II (manufacturing and assembly) of the mmLiVBA project is successful, the improvements may lead to a reduction of the overall vehicle weight of approximately 10 percent and a reduction of approximately seven percent in fuel consumption (although in reality, gas consumption would be reduced by five percent as it is expected that additional accessories will be brought to the vehicle, bringing more weight to other parts of the automobile). Initially, the technologies may be implemented in one car line. If implemented, estimates are that consumers would save $2.65M between 2010 and 2014, and that the new materials would save 6,094 metric tonnes of CO2 eq, when combining reductions of releases of the vehicles and the refineries.45

Within ten years after launch, with the assumption that the technology will be used on three car models, the economic benefits could potentially reach approximately $10M for Canadian automobile owners, in addition to the environmental benefits.46

T&I Transportation
Findings:

  • The Program generated knowledge in the areas in which it conducted research.
  • One technology focussed project has the potential for economic and environmental benefits if widely adopted.

The Program succeeded in developing knowledge in three of its four theme areas (i.e., Vehicle Materials and Design Efficiencies; Advanced, Efficient Powertrains and Fuels; and Support to Policy Development and Integration)47 which expanded the knowledge base of transportation’s contribution to climate change. These include characterizing GHG emissions from vehicles with diesel engines, off-road vehicles, and vehicles with advanced emission control technologies. The Program has provided data to Environment Canada’s emissions inventory group so that the emission factors may be updated.48

The Program also generated a substantial amount of knowledge with respect to black carbon’s (BC’s) role in climate change, such as scientific advances related to BC measurement, databases/inventories and models. It is anticipated that knowledge generated by the Black Carbon project will contribute to improved policy development and assessment, as the enhanced modelling capability will permit greater confidence in the results obtained from an assessment of the effects of different codes, standards, and policy or program choices on climate change.

With respect to advancing the development and implementation of promising new technologies, the Program generated the following outputs:

  • two new high temperature aluminum alloys that are expected to tolerate the harsh conditions of a diesel engine;
  • initiation of an ambitious and major multilateral technology project to develop a magnesium automobile front-end (jointly funded with the CLiMRI Program);
  • development of a composite material with nano-reinforcement which was applied to aluminum brake rotors (jointly funded with CLiMRI); and
  • development of novel electrolytes that are both safer and less expensive compared to existing electrolytes, which meet many of the performance characteristics for usage in lithium ion batteries.

Economic Impacts
An example of the potential economic impacts R&D can have is illustrated by the Reduction of Aerodynamic Drag project. This project was designed to test market-ready components that can be added to trucks to reduce drag, improve fuel efficiency and determine which combinations of components result in the greatest drag reduction. Overall the best combination of the various components (tractor-mounted gap sealing, trailer side skirts and trailer boat-tailing) provided a total drag reduction of 7.9 percent.49

To date, 110 truck trailers have adopted these components, which will – in the most pessimistic scenario – result in $0.8M in savings over five years. This will also result in 800,000 litres of fuel and 2,160 metric tonnes CO2 equivalent in savings over the same five year time period.50

HEE
Findings:

  • The HEE Program advanced the state of knowledge in the areas of hydrogen production, storage and utilization.
  • Two major successes of the Program were: (a) its contribution to the development and publication of the Canadian Hydrogen Installation Code (CHIC); and, (b) development of the world’s first 700-bar hydrogen storage cylinder.

One focus of the Program has been to advance the development of fuel cell technologies. The HEE Program funded approximately 18 fuel cell-related projects from 2003-04 to 2006-07. The two largest funders of these projects were NRCan and industry, providing $9M and $9.2M, respectively.51

NRCan and HEE supported the development of the Ballard fuel cell as well as the Hydrogenics fuel cell. Hydrogenics and Ballard are actively engaged in the development, commercialization, marketing and distribution of fuel cell solutions for forklifts which is reported to be a $5 billion market.52 Hydrogenics has demonstrated its fuel cell for forklifts at General Motors of Canada’s Car Assembly plant and FedEx’s logistics hub at the Pearson International Airport. The Ballard fuel cell technology has been demonstrated in Cellex Power Products Inc., forklifts which were used in Wal-Mart. There are currently (fall 2008) 300 Canadian-built fuel cell fork lifts in operation with an estimated 900 additional fork lifts in operation by the end of 2008.

Before hydrogen and fuel cell products are commercialized and widely available to the general public, the technology will be transferred to early adopters including backup power and materials handling applications such as forklifts.55 Recently, Canadian companies such as Ballard Power Systems and Hydrogenics Corporation signed long-term fuel cell supply agreements with backup power technology providers.56

The HEE also supported fuel cell engine development. Both Ballard and Hydrogenics have had projects funded by HEE that have contributed to advancing the state of the industry. Hydrogenics developed a fuel cell and integrated it into a hybrid electric transit bus which was successfully demonstrated in Winnipeg, Manitoba. Completion of this project led to further work in the area, including two projects funded by the United States Federal Transit Authority.

The research conducted by HEE contributed to the development of the Ballard fuel cell engine, which is part of the first demonstration of fuel cell electric vehicles in Canada.57 The Vancouver Fuel Cell Vehicle Program is testing and evaluating five Ford Focus fuel cell electric vehicles for a period of three years. The vehicles use Ballard fuel cell engines and Dynetek hydrogen storage tanks.58 Both of these products were supported by HEE. Interviewees also reported that the Ballard fuel cell will be used in 20 BC Transit buses which are scheduled to be in place for the 2010 Olympics.

Specific projects that HEE conducted that contributed to development of new technologies are described below by technology area:

  1. Storage
    One of the Program’s major successes was its contribution to the development of the 700-bar aluminum line hydrogen storage cylinder. This project, along with two other NRCan-assisted projects was instrumental in Dynetek’s development of high pressure, hydrogen storage cylinders for vehicle use. The storage capability provided by this project was further developed for specific customer applications, and with additional customer funding. Dynetek was the first manufacturer to gain 700-bar certification for vehicle use and has the most cylinders in service for fuel cell vehicles (FCV) applications worldwide. According to interview and case study data, approximately 200 cylinders for 700-bar use have been supplied.59
     
  2. Utilization
    The HEE contributed to the development of an advanced technology on portable fuel cell power systems for military applications. This four-year project improved the performance of the system by 25 to 50 percent and reduced its size. Integration of the fuel cell auxiliary power unit (APU)into a military vehicle and validation of hydrogen/safety storage are well underway.

    The HEE also contributed to the development of Angstrom Power Incorporated’s proprietary fuel cell architecture. The work progressed from initial fundamental development through a series of design and prototyping activities. The company is now commercializing portable fuel cell power sources built-in to devices such as flashlights and two-way radios.

  3. Production
    Direct Hydrogen Gasification Project60
    Based on earlier work completed on the first generation Direct Hydrogen Gasification (DHG), a company led a project consortium to develop a second generation hydrogen priority gasification process based on the steam-oxidation of iron.

    The project was described as having turned science principles into an engineered prototype. The prototype has the potential for a positive impact on GHG emissions, assuming that carbon sequestration will be widely available. There are a number of steps that need to be completed before the technology has the potential to be commercialized (e.g., refining the prototype design, scaling up the prototype for demonstration, and constructing a full-scale facility).

  4. Codes and standards implemented (e.g., Hydrogen Installation Code) and input to energy policy
    A major barrier to the adoption of hydrogen technologies in Canada has been the lack of acceptable codes and standards to govern the basic safety and operability requirements of hydrogen technologies.61 In order to address this barrier, the Hydrogen Energy Economy Program worked in collaboration with NRCan’s Canadian Transportation Fuel Cell Alliance, the Hydrogen University Safety Network and the Bureau de normalisation du Québec (BNQ) to develop the Canadian Hydrogen Installation Code (CHIC).62 This code was published in January 2007 making Canada the first country in the world to adopt a hydrogen installation code.63

    In addition to the CHIC, the HEE worked with a partner to publish the Guide for Basic Hydrogen Safety and the Society of Automotive Engineers’ (SAE’s) standard for hydrogen fuelling connectors as an international standard.

T&I Demonstration: TEAM transportation projects
The following section describes four TEAM demonstration transportation projects that were covered by evaluation case studies: a) Home Natural Gas Refuelling Appliance; b) High Pressure Compressed Hydrogen Fuelling System for Hydrogen Fuelling; c) Diesel Electric Propulsion for Fishing Vessels; and d) Intelligent Control Systems for Fuel Cell and Natural Gas Vehicles.

The projects show that advancements have been made in the development of more efficient transportation technologies. The technologies, if fully developed and commercialised, could contribute to cost savings, increased economic activity and environmental benefits such as reduced GHG emissions.

  1. Home Natural Gas Refuelling Appliance
    A $9M demonstration project helped a Toronto-based company develop a Home Refuelling Appliance (HRA) for Natural Gas Vehicles (NGVs). The company now sells the HRAs around the world. When compared to gasoline fuelled vehicles, NGVs produce 25 percent less GHG emissions, 70percent less carbon monoxide, 87 percent less nitrogen oxide, and 20 percent less carbon dioxide. TEAM provided $1M and leveraged 7.7 times the funding from other sources.64
     
  2. Intelligent Control Systems for Fuel Cell and Natural Gas Vehicles This project developed enabling technologies for fuel cell vehicles (FCVs) and natural gas vehicles (NGVs), in order to reduce their costs in comparison to conventional vehicles, with the ultimate goal of reducing greenhouse gas (GHG) and other emissions. The project focused on developing intelligent fuel control systems for NGVs, as well as electronic gas pressure regulators, and diagnostic systems for high-pressure storage tanks, for use in both NGVs and FCVs.

    TEAM provided funding of $0.8M from 1999 to 2002 and leveraged 2.2 times the funding from other sources. While the technological development was successful and resulted in five patents being granted, market growth of NGVs & FCVs has not materialized in North America and is not expected to in the near future. As a result, the focus has shifted to FCVs and to international opportunities for NGVs.

    The TEAM Program also developed the System for Measurement & Reporting of Technologies (SMART) process to determine the GHG emission reductions achieved in demonstration projects and applied this methodology to the transportation demonstration projects funded during this period. TEAM developed detailed GHG emission reduction forecasts for each project funded that could be used as reference documents for future transportation demonstration projects. The Program developed a sector-specific protocol called "Fuel Cell Transportation Sector Specific protocol" which can be generally applied to such projects. The TEAM SMART process was submitted to a United Nations ISO standards committee and forms the basis of a new, international standard (#14064 part 2)65, which now applies to any GHG emission reducing project.

  3. Diesel Electric Propulsion for Fishing Vessels
    The purpose of this project was to develop a commercially viable diesel-electric propulsion system for medium sized fishing vessels, to determine how this system would affect the costs of operation, and to identify its impact on greenhouse gas (GHG) emissions. The project began in 2004 with $1.2M in TEAM funding and a total budget of $12.6M. It was halted at the installation phase in March 2008 due to the major downturn in the fishing industry. As a result, the project has not commenced the operational testing and analysis phase. If the project recommences and is successful, the project will develop a Canadian technology with significant environmental benefits that can have potential global application.
     
  4. High Pressure Compressed Hydrogen Fuelling System for Hydrogen Fuelling This project focused on designing, manufacturing and demonstrating a high-pressure hydrogen compressor to permit self-service fuelling for cars and buses. If successful, this innovation will allow for a 20-30 percent reduction in life cycle costs of the hydrogen and will contribute to increased vehicle range, which is currently a major gap in the technology. TEAM provided $1.9M towards the project, which began in 2003-04 and ends in 2009-10. If the company is successful in developing and commercializing the compressor, the compressor will contribute to the development of the distribution systems necessary to improve the economics associated with the delivery of compressed hydrogen. This should also support the future adoption of fuel cell vehicles.

CTFCA
Findings:

  • The Program successfully showcased refuelling demonstration projects; evaluated various hydrogen fuelled vehicles for commercial and private use; and delivered the national supporting framework to enable the development of the fuelling infrastructure such as technical codes and standards, training, certification and safety.
  • The CTFCA contributed to the development of 11 permanent (and four currently in the development phase) hydrogen fuelling66 stations in Ottawa, Toronto, British Columbia, Saskatchewan, Montreal and Prince Edward Island.
  • The CTFCA successfully demonstrated 60 fuel-cell powered vehicles, some of which were dual-fuelled by gasoline or diesel as well as fuel cells. Five of the 60 were fuel cell- powered passenger cars, 19 were fuel cell powered fork lifts, one a baggage tugger, three were dual fuel pickup trucks, 30 were buses and two were vans/sports utility vehicles.
  • The CTFCA provided advice to industry and government on the most viable hydrogen pathways, the circumstances that impact this viability, and the conditions in each region influencing the viability.
  • The CTFCA established some of the basic national policy instruments needed to support implementation of hydrogen as a power currency and contributed to international work in the same area.

The Light-Duty Vehicle Fuel Demonstration Working Group developed, demonstrated and assessed fuelling systems for light-duty fuel cell vehicles. Partners involved in these projects delivered technologies that are currently being tested for commercial application. A three-year test of five passenger cars powered by fuel cells, with hydrogen refuelling stations in Vancouver and Victoria, was reported to have demonstrated that these technologies hold solid promise for the transportation sector.67

Hydrogen fuelling projects were also conducted by the Heavy-Duty Vehicle Fuel Demonstrations Working Group. The projects focused on installation and demonstration of fuelling systems for heavy and medium duty fuel cell vehicles such as: testing shuttle buses in Prince Edward Island68; analyzing the use of off-peak nuclear power for producing hydrogen for a small fleet of intra and inter-urban buses; and the plan to put 20 fuel cell buses on the road in Whistler, British Columbia for the 2010 Winter Olympic and Para Olympic Games.69 In addition, this working group supported the delivery of a number of feasibility studies, the majority of them aimed at determining whether cost-effective fuel cell-powered vehicles (buses) could be delivered and operated. The results indicated that this is possible.

Monitoring the resulting greenhouse gas emissions from projects was listed as a sub-objective of the Program, and the Program is now (June 2009) compiling and analysing that data. The data were not available at the time of this evaluation. It should be noted that demonstration projects were selected in part because of the impact they could have on reducing GHG emissions based on the hydrogen pathways that were modeled by NRCan’s GHGenius model.

The Codes and Standards Working Group (CSWG) was responsible for “developing the necessary supporting framework for the fuelling infrastructure, including technical codes and standards, training, certification and safety.” The CSWG’s mission was to reduce barriers in the areas of codes and standards and in training and certification of personnel that could impede demonstration projects. There were twelve CTFCA-funded projects involved in establishing codes and standards, seven of which were undertaken to address regulatory challenges associated with installing hydrogen equipment in Canada; specifically, eliminating market access barriers to commercialization of fuel cell and hydrogen technologies.70

The CSWG assisted in the development of nationally and internationally-accepted standards for hydrogen related technologies. One of the major achievements of this group was the development of the Canadian Hydrogen Installation Code (CHIC). The CHIC provides installation requirements for a) hydrogen generating equipment; b) hydrogen utilizing equipment; c) hydrogen dispensing equipment; d) hydrogen storage containers; e) hydrogen piping systems; and f) all related accessories. This code established a common understanding and baseline for working with hydrogen related technologies.71

Interviewees expressed the belief that by establishing the CHIC, Canada had international influence on codes and standards through the International Organization of Standardization (ISO) and similar agencies. The CHIC is also expected to help harmonize codes with the US. The CHIC is not yet approved by the provincial regulators and further work on the safe distance for clearance and for explosion proof exclusion is needed.

Other achievements related to codes and standards include an update of the Canadian Electrical Code for Hydrogen, and a permitting guide for installations in Canada (clarifying approvals processes for hydrogen and fuel cell projects in Canada).72 Additional work in this area encompassed overseeing testing of fuelling station and indoor vehicle storage location safety equipment, as well as identifying personnel categories requiring certification.73

The Study and Assessments Working Group supported the CTFCA’s committees and other working groups by developing a compendium of technical, socio-economic and environmental studies regarding fuelling pathways. In addition, it published a series of technical and economic evaluations of different fuelling systems for fuel cell vehicles and the associated emissions performance. These documents are expected to serve as benchmarks for future hydrogen fuel cell technology research.74

A Communications Working Group was charged with communicating the mandate, objectives and activities of the CTFCA and its partners’ activities in the fuel cell and hydrogen industry. During the CTFCA’s seven years, this Group completed projects aimed at raising awareness of the need and role of the hydrogen fuelling infrastructure. The Group also updated stakeholders on the progress of CTFCA projects and their contribution to the future commercialization of fuel cell and hydrogen technologies.75


35 Black carbon is also known also as elemental carbon and soot carbon. L.A. Currie, J.D. Kessler, and Contributors, Isotopic Black Carbon in the Environment: New Metrology for 14C and its International Impact, National Institute of Science and Technology, paper 7149 (Aug. 1999). Source: http://www.cstl.nist.gov/div837/Division/techac/1999/IsotopicBlackCarbon.htm, accessed June 3, 2009
36CO2 eq. is a unit of measure used to allow the addition of or the comparison between gases that have different global warming potentials. Source: http://www.ec.gc.ca/indicateurs-indicators/default.asp?lang=En&n=54C113A2-1#glossaryc, accessed on July 14, 2009.
37 Source: http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=38700, accessed on June 17, 2009
38 Interview and SED, NRCan Transportation S&T Case Studies (July 8, 2008).
39 Office of Energy Research and Development, NRCan, CLiMRI Plan at the Objective Level (Program) Objective 2.2.1 Program Annual Report (PAR) 2006-2007 (April 20, 2007), page 35.
40 Interview data and Office of Energy Research and Development, NRCan, CLiMRI Plan at the Objective Level (Program) Objective 2.2.1 Program Annual Report (PAR) 2006-2007 (April 20, 2007), pages 38-39.
41 Interview data and SED, NRCan Transportation S&T Case Studies (July 8, 2008).
42NRCan Strategic Evaluation Division Transportation S&T Case Studies (July 8, 2008).
43CLiMRI, Canadian Lightweight Materials Research Initiative (CLiMRI) 2006-2007 Annual Report Plan at the Objective Level (April 21, 2006), page 60.
44 The USAMP is an industry partnership that began in 1993, comprised of GM, Ford and DaimlerChrysler. It is aimed at, “developing materials and processes that enable the high volume production of vehicles that when compared to today’s vehicles are: half the mass as affordable more recyclable of equal or better quality and durability.” (source: http://www.uscar.org/guest/view_team.php?teams_id=28)
45 SED, NRCan Transportation S&T Case Studies (July 8, 2008).
46 SED, NRCan Transportation S&T Case Studies (July 8, 2008).
47 Two of the four T&I themes were relatively inactive. Although the Advanced, Efficient Powertrains and Fuels theme generated knowledge, it only conducted two projects. The other inactive theme, Intelligent & Efficient Transportation Systems, conducted one project.
48
CCTII: Advanced End Use Efficiency: Transportation (TIB6): 2006-07 Annual Report. Environment Canada reports that Emission Factors (EF) can be used to estimate the rate at which a pollutant is released into the atmosphere (or captured) as a result of some process activity or unit throughput. Source: http://www.ec.gc.ca/pdb/ghg/guidance/calcu_fac_e.cfm
49T&I Transportation Program. Annual Report 2005-06.
50 SED, NRCan Transportation S&T Case Studies (July 2008)
51HEE Project financial database.
52 Hydrogen and Fuel Cells Canada, Government of Canada (Foreign Affairs and International Trade Canada, National Research Council and Natural Resources Canada), Government of British Columbia, Canada’s Hydrogen + Fuel Cells Industry Capabilities Guide 2007 (2007), page 21.
53 Ibid page 21.
54 Ibid page 20.
55 Ibid page 20.
56 Ibid page 17.
57 A three-year, $8.7M project involving the Government of Canada, Hydrogen & Fuel Cells Canada, Ford Motor Company, and the Government of British Columbia per Hydrogen and Fuel Cells Canada, Government of Canada (Foreign Affairs and International Trade Canada, National Research Council and Natural Resources Canada), Government of British Columbia, Canada’s Hydrogen + Fuel Cells Industry Capabilities Guide 2007 (2007), page 23.
58 Ibid.
59 Interview data and SED, NRCan Transportation S&T Case Studies (July 8, 2008).
60 Project Review and Interview data.
61 National Round Table on the Environment and the Economy, Case Study on the Role of Fiscal Policy in Hydrogen Development – Economic Analysis, (May 2004), page 19.
62 Canadian Transportation Fuel Cell Alliance, Fuelling the Drive Annual Report 2003-2004 (undated), page 8.
63 Canadian Transportation Fuel Cell Alliance, Fuelling the Drive Final Report 2001-2008 (undated), page 25.
64 SED, NRCan Transportation S&T Case Studies (July 8, 2008).
65 source: http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=38700, accessed on June 17, 2009
66 Program director, presentation to Roads2HyCom Workshop, Hydrogen and Fuel Cell Research, Development and Demonstration, Athens Greece (November 2008), slides 7-13.
67 Interview, case study and document review data, including: The Canadian Transportation Fuel Cell Alliance, Fuelling The Drive Final Report 2001-2008 (undated), pages 25-27.
68CTFCA HD/LD and Fuelling Demo Working Groups, Wind-Hydrogen Solutions PEI Projects Update Presentation made in Calgary, Alberta (January 29, 2008), slides 5-9.
69 SED, NRCan Transportation S&T Case Studies (July 2008, pages 128, 137 and 140).
70 Interview data
71 Bureau de normalisation du Quebec press release NEW ON THE SHELF: CANADIAN HYDROGEN INSTALLATION CODE (CHIC) (July 3, 2007). Source : http://www.bnq.qc.ca/documents/communique_bnq-chic-rncan_en.pdf
72 Industry Canada, News and Events 2007 Source: http://www.h2fcfuture.gc.ca/news/index-eng.htm and interview data.
73 Codes and Standards Canada, Codes & Standards Canada, Federal Regulations, Canadian National Standards CAN/BNQ 1784-000 National Standards of Canada Canadian Hydrogen Installation Code Source: http://www.fuelcellstandards.com/2.2.htm.
74 Examples include: Tax Policy Services Group of Ernst & Young for NRCan, An Economic Analysis of Various Hydrogen Fuelling Pathways from a Canadian Perspective (October 29, 2003), page 11; Greenleaf Integrated Energy Systems Inc., ZEDH2 Feasibility Study (May 2005), ICF Consulting Inc Greenhouse Gas and Cost Impacts of Canadian Electric Markets with Regional Hydrogen Production (March 2004).
75CTFCA, Fuelling the Drive Final Report 2001-2008 (undated), page 12.

4.2.2 To what extent did the Programs influence the constructive engagement and collaborative networks to further research, development and deployment activities?

Fulfilling the various objectives of the Programs that comprise the Transportation Sub-sub Activity requires not only research, but effective networking and technology transfer. This part of the success analysis focuses on the extent to which the Programs established successful networks to achieve their objectives.

The Programs influenced the engagement of stakeholders and collaborative networks to further research, development and demonstrations. They developed interdepartmental participation in conducting projects, managing the Programs, and sharing information at the project, Program and departmental level. There was also technical and financial collaboration within projects among other government departments, industry, universities, provinces, and organizations in other countries.

Overall, the Programs were generally effective in bringing together a diverse group of organisations to further RD&D goals. However, there is limited information on the research and development Programs’ levels of engagement (this does not include CTFCA and TEAM since they are demonstration programs) with the client groups who are required to adopt and act on the research results for an impact to be realized. The transfer, uptake and implementation of research results by client groups are essential to achieving impacts. Although collaboration among departments worked well, increased participation from industry and policy-makers is needed since researchers depend on industry for information about technologies and samples for their research as well as information from the policy and regulatory community on the direction of government policy. It should be noted that the timing involved with respect to providing input among the R&D, industry, and policy communities does not always align well. For example, policy input windows of opportunity may be relatively short compared to the time required to carry out quality R&D work.

Between 2001-02 and 2007-08, OERD provided research funds to six federal government departments/agencies for transportation S&T R&D, as indicated in Table 3. The distribution of total OERD/PERD funding to recipient departments during that period was approximately $57.2M. CANMET of NRCan received approximately $36.4M or 64 percent of the total funding provided by OERD/PERD.

Table 3:
Estimates of PERD/NRCan Funding to Transportation S&T Recipient Organizations, 2001-02 to 2007-08, ($K)
Organization $ %
NRCan/CANMET/CETC 36,425 64
NRC 6,228 11
EC 5,906 10
HC 3,222 6
DND 2,770 5
TC 2,647 5
Total 57,198 100

Note: This table includes information for the following: a) Particulate Matter (2001-02 to 2007-08); b) AFTER (2001-02 to 2007-08); c) CLiMRI (2001-02 to 2007-08); d) HEE-PERD portion (2001-02 to 2007-08).
Source: OERD financial records contained the overall PERD expenditure by department for all Programs and the evaluation used these records as the primary source of information.

Table 4 shows that between 2001-02 to 2007-08, $15.6M of CCTII (& CCAP) funding was distributed to eight recipient departments.

Table 4:
Estimates of CCTII (&CCAP)/NRCan Funding to Transportation S&T Recipient Organizations, 2001-02 to 2007-08, ($K)
Organization $ %
NRCan/CETC/CANMET 10,497 67.2
NRC 2,708 17.3
EC 1,476 9.5
DND 660 4.2
TC 201 1.3
Western Economic Diversification 20 0.1
IC 25 0.2
DFAIT 22 0.1
Total 15,609 100

Note: This table includes information for the following: a) Hydrogen Energy Economy: T&I Portion (2003-04 to 2007-08); b) T&I Transportation (2003-04 to 2007-08); and c) CTFCA (2001-02 to 2007-08).
Source: A) OERD financial records contained the overall T&I expenditure by department for all Programs and the evaluation used these records as the primary source of information. B) CTFCA was managed separately by NRCan’s CETC (CANMET Energy Technology Centre) and it tracked the funding received from CCTII and CCAP. The CTFCA’s portion is approximate because the Program’s financial records were based on budget commitments and expenditures.

From 2001-02 to 2007-08, the combined PERD and CCTII (& CCAP) funding contributions were $72.8M and were distributed to eight participating organizations. For the details by each Program see Annex C.

Recipient departments contributed A-base resources to Transportation S&T Sub-sub activity and may have leveraged funding from other sources such as other PERD Programs, and/or other federal government and external sources. Figures provided in Program Annual Reports indicate that on average, between fiscal year 2002-03 and 2006-07, roughly $29.1M (15 percent) of the programs’ annual funding came from other government departments. Partner contributions from Non-Government of Canada sources, such as industry, universities, NGOs, provincial and territorial governments and international, were approximately $75.8M (38 percent) for the same period. For estimates of funding contributions from partner organisations for each program, see Annex D.

Funding multiple departments allows for an interdepartmental, and hence multi-disciplinary, approach to address the research issues. Given that other departments contributed A-base resources and were able to leverage other sources of project funding suggests that these parties shared a common interest in the research. Several interviewees commented that a major strength of the programs is that they built interdepartmental teams that collaborate on common research themes. The multidisciplinary nature of the programs assists in developing working relationships and reduces silos.

There was also scientific, technical and financial collaboration at the project level among government departments and agencies in Canada and abroad, universities, S&T organizations, provincial organizations, and members of the petroleum and engine manufacturing industry. The evaluation case studies indicate that ‘collaboration’ encompasses many activities including sharing resources (labs, equipment, expertise); sharing data and information; sharing/providing samples (fuels, collected emissions) or technologies for research purposes (e.g., catalysts, engines); jointly conducting research; and testing/evaluating research products (e.g., prototypes).

The general findings on the extent to which the programs developed collaborative networks to further RD&D activities are presented below.

AFTER
Finding: The Program has created multi-departmental participation in conducting research projects, managing the Program, and sharing information. However, there is limited information on the Program’s level of engagement with the client groups who are required to adopt and act on the research results in order for impacts to be realized.

The Program was able to obtain participation from key players needed to conduct relevant research, particularly other departments, and this has increased over time. Some interviewees commented that the Program needed more involvement from industry which would have helped researchers access information on engines and fuel samples for toxicity testing.

The multi-departmental approach to conducting the research was facilitated by the Program Management Committee which consisted of departmental representatives from four departments (two members from NRCan, and one each from HC, TC, and EC), and one member from the Canadian Petroleum Products Institute. The Program had annual review meetings, during which researchers presented the progress of their projects. These meetings created a forum for sharing information, helped develop networks among the program participants, and facilitated the coordination of research projects.

In terms of formal inter-program communication, a joint Particles-AFTER Program information meeting was held in June, 2006 to stimulate existing and new cross- program and external collaboration and promote early exchange of information that could influence research directions. The meeting provided opportunities for one-on-one discussions about cross-Program activities that had not taken place in any other fora. One issue that surfaced during the meeting was the need to hold additional discussions with Canadian policy and regulatory decision-makers in 2006-07 to discuss uptake of results.76

Finally, the AFTER Program co-sponsored the Windsor workshop with the U.S. Department of Energy, as well as joint Canada/U.S. workshops led by the AFTER and the PERD Oil Sands Upgrading Program (held in 2005 and 2007). Several interviewees commented that the Windsor Workshop was well attended by industry and there was a strong exchange of ideas on technological issues and future directions in the industry.

Particulate Program
Finding: The Program has influenced the engagement of stakeholders and collaborative networks to further research and development. However, if the tools and knowledge developed by the Program are to influence forward thinking on standards and policies, the Program’s linkages with its policy and regulatory client groups will have to be strengthened.

The current evaluation has similar findings to the 2003 evaluation with regard to the engagement of stakeholders (policy makers, regulators and clients/partners). Engagement has largely been on a case-by-case basis, rather than program or industry-by-industry basis. This is probably reflective of the Program’s activities which remain focused on improving the understanding of the formation, transportation, and health effects of particulate matter, and also because of its relatively small level of funding. To date, the lack of formal relationships with the policy and regulatory client communities has not been a major issue. However, moving into the future, if the knowledge and tools developed in this Program are to be more strongly applied, such relationships, and two-way communications in them, will be increasingly important.77

The Program would benefit from more stakeholder involvement in program decision-making, by means of an active committee or similar mechanism, reflecting the increasing maturity of the Program’s results. An active committee would provide advice on strategy and operational issues such as targeting and deployment to end-users.

According to the current Program Plan, as well as two Annual Reports for 2005-06 and 2006-07, an External Advisory Committee existed. However, the evaluation found that the Committee was not active and played no role in the Program. The Program Plan and the Annual Reports list seven members representing two universities, Environment Canada, NRCan, and one private company. The Program Plan explains that the Committee “represents the broader science, policy and regulatory communities within the federal government and beyond. The External Advisory Committee provides advice to the Program Leader and the Program Management Committee on broad directions for their research, and may provide peer review comments on specific Program projects.”78

The Program has created opportunities for multi-departmental participation in conducting research, managing the Program, and sharing information at the project, Program and departmental level. There were also technical and financial collaboration within projects other government departments, industry, universities, provinces, and organizations in other countries.

In addition to the multi-departmental approach to conducting the research, a new Program Management Committee was created for the 2005-06 to 2008-09 funding cycle. The new Committee consists of departmental representatives from five departments that are not directly involved in the projects funded by the Program. This is in contrast to the Committee membership in the first cycle, which consisted of recipients of PERD project funding. Instead, the new Management Committee members are policy advisors, risk assessors, managers, and a research scientist. The departmental representation for the new Committee includes NRCan and TC, which were not represented in the first cycle.79

The Program had annual review meetings during which researchers presented the progress of their projects. Interviewees commented that these meetings were very helpful in terms of sharing information among researchers and served to create a knowledge base and contacts in other departments. A number of interviewees felt, however, that there was poor communication between scientists and the Management Committee during the meetings. Two interviewees suggested that there may be a lack of interest on the part of the policy community because the meetings address long-term scientific outcomes and policy makers are more interested in the short-term policy implications.

Finally, as discussed under the AFTER Program, a joint Particulate Matter-AFTER Program information meeting was held in June, 2006.80

CLiMRI Findings:

  • CLiMRI has been successful in developing domestic and multilateral partnerships.
  • Industry connections are extremely important to the development of technology in the automotive sector. As a result, it is important that partners for research projects are identified at the time of R&D proposal development (i.e., in advance of R&D work starting) to facilitate technology transfer.
  • Limited funding, proprietary interests and intellectual property policy were described as partnership/network limitations by interviewees.

According to interview and documentation, CLiMRI’s major success is the collaboration and support it received from industry.81 The Program established alliances and partnerships with three major North American vehicle manufacturers (and several in Europe), more than ten North American universities, 37 companies (auto part manufacturers, material suppliers, etc.), six federal labs and participated in two major multi-million dollar USAMP programs. This facilitated the transfer of knowledge to Canadian suppliers and manufacturers. CLiMRI also helped establish collaborations with the governments of other countries (U.S. Department of Energy and China) as well as laboratories in Asia, Europe and Australia.82

A significant factor in the Program’s networking effectiveness was the involvement of NRCan’s CANMET Materials Technology Lab (MTL) as manager of the CLiMRI Program, due to its strong knowledge base and facilities for handling advanced light-weighting materials as well as its good historical working relationships with companies involved in materials (such networks require years of effort to build).

The Program’s Industry Steering Committee (ISC) is important in obtaining industry perspectives, trends, support and feedback, and is the nexus of a significant network in and of itself. The ISC is composed of private sector representatives, such as automotive manufacturers and metal companies, automotive component makers, research networks (such at Auto 21) and a member of the U.S.DOE.83 The ISC provides guidance on major technical thrusts and challenges, and helps steer projects.84

Even given these networking achievements, interviewees noted that MTL/CLiMRI needs to improve its performance in bringing international players to the table and marketing their expertise. CLiMRI does not have the business development expertise to approach large businesses (CLiMRI contains materials experts, not business management experts).

Intellectual property (IP) policies were described as also imposing limits on partnerships and a barrier to working with government by industry. Interviewees indicated that industry would like to participate more in CLiMRI, and to develop larger projects, if NRCan eased its limitations on IP. As a result of this limitation, CLiMRI industry members address technologies in a general sense, in order to protect proprietary interests.

Industry looks to MTL/CLiMRI to bring a technology or process to a prototype stage and to develop the knowledge and product desired. However, industry does not want to transfer all of its knowledge to government because this could result in loss of core technologies and manufacturing knowledge. Industry rarely discloses trade secrets when participating in CLiMRI partnerships in order to maintain competitiveness. This imposes limits on CLiMRI partnerships.85

It was noted by interviewees that MTL/CLiMRI may be relying too heavily on partnerships for leveraging funds. Because the Program’s budget is relatively small, CLiMRI is restricted in its ability to develop “bigger and bolder” impact technologies. CLiMRI initiates numerous smaller projects and uses its expertise to lever partnership funding. This is not always successful, unless projects are well planned prior to receiving approval so that partnerships can be established. For example, a project successfully developed a technology, but without a partner to uptake the results. As a result, the technology has not been implemented.

Another partnership issue identified in interviews was that CLiMRI runs the risk of undertaking projects that are too company-driven in that they may convey a marketplace competitive advantage to one specific company. For example, CLiMRI developed an all aluminum diesel engine block and head for one specific company. The company will have an economic advantage being the only company in North America to cast and produce aluminum engines and compete with European companies that already have the technology.86 While company specific projects might appear to have limited benefit to the public, since the production of advanced engines requires inputs from many smaller firms (tooling, coatings, heat treatment, services, etc.), there could be broader impacts in the supply chain and in the local economy where the production plant is located.

T&I Transportation
Finding: T&I Transportation was able to take advantage of existing networks in PERD and develop collaborations to further research specific to its own goals.

The T&I Transportation Program was designed with the intent to take advantage of the existing PERD research infrastructure while expanding the S&T to specifically address greenhouse gas reductions. In order to do this, the Program established four research themes that were based on existing PERD Programs - CLiMRI, AFTER, Transportation Systems (terminated in 2003-04), and Particles. As a result, many of the participants in the T&I Transportation Program were the same researchers in the PERD Programs. The Program included researchers from four government departments, and eleven branches within those departments:

  • Natural Resources Canada – MTL, Office of Energy Efficiency;
  • National Research Council – Institute for Chemical Process and Environmental Technology (ICPET), Industrial Materials Institute (IMI), and AL;
  • Environment Canada – (Emissions Research and Measurement Division (ERMD), Meteorological Service of Canada (MSC), ACS, Environmental Protection Service (EPS)- Environmental Technology Advancement Directorate (ETAD); and
  • Transport Canada.

Despite this broad participation, the number of participants from Transport Canada was relatively small. Nonetheless, T&I Transportation’s close link with the PERD Programs is emphasized by the $2.7M of funding leveraged from PERD, compared to funding from CCTII of $6M from 2003-04 to 2007-08.

An example of how the T&I Transportation Program took advantage of the existing PERD infrastructure and expertise is illustrated by the Advanced Fuels for homogeneous charge compression ignition (HCCI) Engines project. This was a multi-year project to develop fundamental knowledge on the effects of fuel properties on HCCI combustion at a variety of engine operating conditions, and to develop a practical method for rating the combustion quality (ignition quality and heat release rate) of HCCI fuels. It is very similar to the PERD AFTER project, Advanced Lean-burn Combustion Technologies for internal combustion engines (ICE), which conducted fundamental research and development on the use of lean-burn combustion technologies within a HCCI engine.

Both projects were led by NRC and included researchers from NRCan’s National Center for Upgrading Technology and the US Department of Energy. Furthermore, the T&I Transportation project used the state-of-the-art facility for studying HCCI engines that had been previously constructed for the AFTER project. By taking advantage of this facility and the existing network, the T&I Transportation project was able to focus on getting research results in a more timely and efficient way.

HEE
Finding: The Program was successful in developing networks to conduct pre-competitive R&D, and for developing internationally compatible codes and standards.

The HEE Program was successful in bringing together the R&D community to create a critical mass of experts by developing collaboration with a spectrum of researchers from academia, industry, federal and provincial governments, and international organizations such as International Organization for Standardization (ISO). The Program reported 49 collaborations in 2003-04 (including companies, industry associations universities and other levels of government); 28 formal collaborations in 2005-06; 28 formal collaborations in 2006-07; and 47 project partners and 12 formal collaborations in 2007-08 (no figures were reported in the 2004-05 Annual Report). Between 2003-04 and 2007-08, HEE organized or assisted in organizing nine workshops and/or conferences including the 12th Canadian Hydrogen Conference in 2003-04.

HEE also helped establish the Hydrogen University Safety Network in 2006-07 which links the hydrogen safety activities of Canadian universities with the automotive industry through AUTO2187 and the codes and standards community. AUTO21 involves researchers from Université de Québec a Trois Rivières (UQTR), University of Victoria, Concordia University, and the University of Calgary.

Over the past twenty years, the Canadian Government has invested roughly $300M in hydrogen research; industry invested roughly one billion dollars, or three times as much.88 Between 2001-02 and 2007-08 the Government of Canada invested roughly $54.4M in the Hydrogen Energy Economy Program, of which NRCan provided roughly $42.8M. This investment was leveraged against an investment of roughly $37.6M from industry, non-government organizations (NGOs), universities, and provincial governments. This represents a leverage ratio of just less than 1 to 0.7. Industry was the largest external contributor at approximately $31.4M, followed by universities at $4M, the provinces at $1.8M, and NGOs at $0.4M.

The industrial participants in the Program were mostly small companies (e.g., Hydrogenics, Ballard). Only one automotive manufacturer was involved in the Program. Interviewees reported that partnerships were essential for the success of projects and that one of the strengths of the Program was its ability to leverage partnerships and funds.

The Hydrogen Technical Advisory Committee (HyTAG) was also a strength of the Program as it provided the views of industry on directions of technology and priorities for research. The Committee is composed of some of the leaders in the hydrogen community from industry, academia and government.

The Canadian government (represented mainly by NRCan, Industry Canada and the National Research Council) participates in a number of international hydrogen-related fora. This participation helps focus and inform R&D programs in Canada. Canada participates in two International Energy Agency Implementation Agreements in this area (Production & Utilization of Hydrogen and Advanced Fuel Cells). Through these agreements, 19 countries share information on pre-competitive R&D.

NRCan also has a Memorandum of Understanding with the U.S. DOE and the California Air Resources Board, which allows for joint projects and information sharing between the two countries. HEE has participated in the annual review of U.S. DOE hydrogen projects and participates in DOE meetings to discuss the future of hydrogen, including current capabilities, and priority areas.

These international fora provide a useful venue for exchanging information, conducting pre-competitive R&D, and for developing internationally compatible codes and standards. The linkages give industry and government the information they need to set R&D priorities, taking into account the technology and gaps, the science behind the technology, and the capabilities of Canadian industry, universities and other researchers, relative to the competition.

CTFCA
Finding: The CTFCA was successful in developing collaboration with a wide section of the hydrogen and fuel cell community concerned with on-road demonstrations and fuelling technologies.

All of the CTFCA’s projects involved collaboration and partnerships among a range of partners. However, three in particular, the Vancouver Fuel Cell Vehicle Program, the British Columbia Hydrogen Highway, and the Hydrogen Village in Toronto, are showcases not only of the technologies involved, but of the success of the partnerships and networking involved. The evaluation case studies found them to have been effective models for coordinating the appropriate stakeholders to achieve greater reach and recognition of hydrogen technologies.

For example, the British Columbia Hydrogen Highway encompasses six sites. One of these sites, the Surrey PowerTech Station, was opened in March 2002 and involved BC Hydro, BOC Gases, British Petroleum, ChevronTexaco, Dynetek, JFE Container Co., Ltd, Shell Hydrogen, and Stuart Energy Systems. This station is now the first 700-bar hydrogen fuelling station in the world. Another example, the Vancouver Fuel Cell Vehicle Program, involved the City of Vancouver, Ballard Power Systems, BC Hydro/Powertech Labs Inc., Hydrogen and Fuel Cells Canada/Province of BC and BC Transit in Victoria.89

The CTFCA established networks among Canadian universities and with international agencies such as the International Energy Agency as well as the federal governments of other countries (Mexico, Japan, and United States). Interviewees noted strong relationships with laboratories such as Sandia Laboratory (US DOE), which provided valuable knowledge on fuel cells.90

It should be noted, however, that interview data indicates that improvements could be made to the effectiveness of the networking. For example, it was reported that if the hydrogen infrastructure is to be fully developed, end-user engagement will need to be improved (e.g., bus, stationary application, and forklift manufactures/users).91

Issues were also identified by interviewees that were believed to hinder the ability of the CTFCA to obtain greater participation from oil and gas companies such as a lack of a Canadian alternative fuels policy, as well as perceived high risk involved in new fuels and the need for more research and development to reduce this risk. Finally, with the price of oil at unprecedented levels (at the time that interviews were being conducted for this evaluation),92 oil and gas companies were seen as having limited short-term motivation to work with CTFCA.

Overall, the evaluation shows the effectiveness of the CTFCA’s performance in having established a technical forum that allowed industry-government (federal, provincial, municipal) collaboration on hydrogen technologies. This forum brought together people from various backgrounds (e.g., universities, government, utilities, consortia and associations, as well as other non-government organizations) to examine industry concerns and needs related to hydrogen fuelling stations.


76AFTER Mid-Year Progress Report 2006/07.
77 Support the Development of Technological and Other Measures to Control and Reduce Emissions of Particulate Matter- Interim Evaluation of the Air Quality –Particles Research Program (2003).
78 Support the Development of Technological and Other Measures to Control and Reduce Emissions of Particulate Matter Program Plan 2005/06-2008/09. Final. December 31, 2005.
79 Support the Development of Technological and Other Measures to Control and Reduce Emissions of Particulate Matter Program. POL Plan 2005/06-2008/09. Final. December 31, 2005.
80 Support the Development of Technological and Other Measures to Control and Reduce Emissions of Particulate Matter Program. Mid-Year Progress Report 2006/07. November 15, 2006.
81 Interview data; and Office of Energy Research and Development, NRCan, Canadian Lightweight Materials Research Initiative (CLiMRI) 2005-2006 Annual Report Plan at the Objective Level (April 21, 2006), pages 50-54; and Office of Energy Research and Development, NRCan, Canadian Lightweight Materials Research Initiative (CLiMRI) 2004-2005 Annual Report Plan at the Objective Level (May 2005), pages 60-61.
82 Interview data; and Office of Energy Research and Development, NRCan, Canadian Lightweight Materials Research Initiative (CLiMRI) 2005-2006 Annual Report Plan at the Objective Level (April 21, 2006), pages 50-54; and Office of Energy Research and Development, NRCan, Canadian Lightweight Materials Research Initiative (CLiMRI) 2004-2005 Annual Report Plan at the Objective Level (May 2005), pages 60-61.
83 Office of Energy Research and Development, NRCan, Canadian Lightweight Materials Research Initiative (CLiMRI) 2004-2005 Annual Report Plan at the Objective Level (May 2005), page v.
84 Interview data.
85 Interview data and SED, NRCan Transportation S&T Case Studies (July 8, 2008).
86 Office of Energy Research and Development, NRCan, CLiMRI 2005-2006 Annual Report Plan at the Objective Level (Program) (April 21, 2006) pages 39-42.
87 AUTO21 is a Network of Centres of Excellence. It is intended to support Canada’s position as a leader in automotive research and development. According to its website (http://auto21.ca/index.php), by partnering the public and private sectors, AUTO21 supports more than 300 researchers across Canada working on 54 auto-related projects in a variety of areas.
88HyFATE, Hydrogen and Fuel Cells, Canadian Technology Progress and Future Opportunities (2007).
89 SED, NRCan Transportation S&T Case Studies (July 8, 2008).
90 Interview data.
91 Interview data.
92 Summer and fall 2008.

4.2.3 What factors (both internal and external) affected the Programs’ performance (e.g., human resources, the economy, policy, etc)?

The performance of the programs was affected by the following internal and external performance factors:

  1. Partnerships/networking
    The involvement of appropriate partners is crucial for the success of each Program and their projects. The collaborations were seen as facilitating technology transfers, enabling the Programs to access funding and human resources, exchange ideas, access samples, gain contacts, and conduct research. For example, the five working groups of the CTFCA and the key partnerships that were formed enabled the Program to keep abreast of hydrogen related issues and to develop and advance projects.

    Interviewees stated that projects that include researchers from multiple departments have resulted in better projects. The PERD approach is considered to be advantageous because it enables a good exchange of information between the Program participants on methodology and research findings for projects that are complementary in nature. The multidisciplinary approach to the research is considered to be a strong attribute because it brings together a variety of expertise for the research.

  2. Engagement of stakeholders
    Achievement of the Programs’ objectives is dependent on uptake and implementation of the knowledge, technologies and processes by decision-makers and industry. For all of the Programs, the evaluations found that collaboration among departments and researchers was working well, but that more participation was needed from industry and the policy and regulatory communities. Some examples are provided below:
    • AFTER and T&I: Researchers depend on industry for information about technologies, and samples of fuels for their research. Interviewees indicated that it was difficult to obtain the participation of industry.
    • CLiMRI: MTL/CLiMRI has come to be known for its expertise in metals and materials and has established strong alliances with industry to cost-share high-risk R&D. The ISC has helped improve CLiMRI’s performance by opening up opportunities to the US and internationally, and encouraged industry to buy-in to CLiMRI produced technology. However, there have still been challenges in obtaining the engagement of the end-users of research results as industry does not want to compromise their competitive advantage.
    • PM: Although the Program’s goal is “to provide knowledge and tools that will support the development of technology and other measures to control and reduce emissions of particulate matter”, the project reviews and case studies found that the projects are not clear on how the research results will affect the development of technology or other measures/policy decisions. This may be because most of the projects are conducting basic research (i.e., improving the understanding of the formation, transportation, and health effects of particulate matter) whose intended audience is other research communities. Once its results are fulsome enough it will have to develop mechanisms to translate those results to decision-makers if it is to have an impact aside from generating knowledge.

      Given the disparity between the timelines in which policy and regulatory decisions are made and the time required to conduct science, a need for more information about the needs of policy makers was also described. One example is the need to know the types of fuels under consideration by policy makers. This information would allow researchers to plan appropriately given the lead time that is needed to generate research results.

  3. Participation in management committee
    Participation on the Programs’ Management Committees for the PERD Programs is voluntary and thus the willingness to participate and the time given by Committee members varied depending on personal interest and available time.
  4. Departmental versus PERD priorities
    PERD relies on researchers in several departments to coordinate and conduct research to achieve its objectives. These researchers have to balance their departmental priorities with PERD priorities, which interviewees felt could be a risk for the Programs.
  5. Policy and strategic direction
    Examples include the following:
    • For the PM Program, a need for strategic direction was identified in interviews noted in the previous evaluation. The structure currently in place has not provided advice on strategy and operational issues such as targeting and deployment of research results. According to interviewees, priorities were primarily determined by the individual research scientists rather than in conjunction with decision-makers and the end-users of the results to meet defined policy/technology needs.
    • A positive factor that enabled the HEE Program to advance its research is the long-term support that the hydrogen area received. As the report of the National Advisory Panel on Sustainable Energy Science and Technology noted, “There is a need for a long-term commitment (minimum of 10 years) by the public and private sectors to focus on energy S&T and fund it in a sustained manner. This commitment is essential given the long lead times required for technological change in the energy economy, and the need to develop and retain the human capital that underpins energy innovation.”93 The HEE Program provided some of the long-term stability required for its partners to plan their shorter term projects (i.e., industry working in niche areas such as portable applications in order to generate revenue).
    • Industry interviewees for the HEE Program expressed the belief that the Program focuses too much on areas of long-term research (i.e., fuel cells), and not enough in areas where there is potential for early adoption such as industrial uses (oil sands) and ICEs. By focusing on applications that are feasible in the short-term and encouraging early adoption, the Program could improve the economic feasibility of the overall hydrogen economy.
    • According to CTFCA and HEE interviewees from industry, policy drives the adoption of all technologies, but the government’s vision for hydrogen and other alternative fuels has not been clearly expressed and this has limited the acceptance of hydrogen as a significant fuel alternative. These interviewees stated that this lack of clarity has resulted in a reduction in investment in research, development and demonstration for hydrogen and fuel cells. Industry and government expenditures on hydrogen and fuel cell RD&D has declined from a peak of $290M in 2003 to $193M in 2006.94
    • Prior to 2005, the CTFCA was limited to funding the development, demonstration and evaluating options for hydrogen fuelling stations and hydrogen fuelled vehicles. In 2005, it expanded its scope to fund fuel cell vehicle acquisitions that use hydrogen as a source of fuel provided the vehicles formed an integral part of a hydrogen-fuelling demonstration project. As well, the Program began to allow internal combustion engines (ICEs) fuelled with hydrogen and hydrogen/ hydrocarbon mixtures to be eligible for funding support. The change in scope allowed CTFCA to increase the availability of fuel cell vehicles for demonstration and road testing.95
    • The Transportation T&I Program duration was short (2003-04 to 2007-08). Interviewees felt that it was ‘sprung’ on the scientists in 2003-04 without any preparation. As a result, the Program did not successfully fund any projects until well into the second year, 2005-06.
  6. Funding and project planning
    Examples include the following:
    • Inconsistencies in annual PERD funding were reported by several interviewees as making it difficult for Programs to manage and plan for long-term R&D projects. For example, project leaders indicated that they did not know how much funding CLiMRI was going to receive at the beginning of each year, which made it difficult to plan for long-term R&D.96
    • CLiMRI funded a large number of small projects. The average budget per project from 2003-04 to 2006-7 was $136K. Evidence indicates that the problem with supporting smaller projects is that it reduces the ability of projects to generate impacts, at least in part because it limits funding and expertise, as well as collaboration among the automotive R&D community.
    • Redirecting funds to more promising projects was a challenge for CLiMRI because the Industry Steering Committee meets only once a year to decide which projects to fund.
    • The Transportation T&I Program had an appropriate balance of work between the objectives of developing and implementing promising new technologies, and improving understanding of the role of the transportation sector in climate change. However, two of the four themes were relatively inactive due to a lack of quality research proposals;97 and therefore, it is unlikely that the Program could have fully achieved its objectives.
  7. Human resources challenges
    Examples include the following:
    • The programs were not able to hire and retain enough research scientists with the required specialized skills needed for project work, and this resulted in some project delays.
    • For the Transportation T&I Program, the Program Manager changed three times in five years.

93 The National Advisory Panel on Sustainable Energy, Science and Technology; Powerful Connections, The Report of the National Advisory Panel on Sustainable Energy, Science and Technology Priorities and Directions in Energy Science and Technology in Canada (2006), page 8, Catalogue No.: M4-40/2006E ISBN: 0-662-43412-9.
94 Government of Canada (Industry Canada), Hydrogen & Fuel Cells Canada and PricewaterhouseCoopers.
95CTFCA. Fuelling the Drive. Program Annual Report 2004-05. (Page 2).
96 Interview data and SED, NRCan Transportation S&T Case Studies (July 8, 2008).
97 The two T&I themes that were relatively inactive: 1) Advanced, Efficient Powertrains and Fuels (2 projects); and 2) Intelligent & Efficient Transportation Systems (1 project).

4.3 Cost Effectiveness and Alternatives

4.3.1 To what extent were the Programs an appropriate and effective means of achieving objectives?

The Transportation Sub-sub Activity was delivered in a cost-effective manner. The programs were generally appropriate and efficient for reaching their objectives. The planning, review and reallocation processes for projects ensured proposals fit under the overall direction and objectives of each program. It should be noted that because of the nature of science and technology research and development, it is difficult to show the cost-effectiveness in relation to the immediate return on investment. Results in science and technology research and development develop over a longer timeframe and require a long-term commitment.

The issue of the cost-effectiveness was examined from three perspectives. The first section provides a discussion of financial leveraging; the second section discusses program efficiency; and the third section examines economic impacts of transportation R&D projects.

Overall Leveraging One of the key methods to assess the cost-effectiveness of a government program is to examine the extent to which the program is capable of leveraging investments from other sources.

As shown in Table 1, the total NRCan funding on the Transportation S&T Sub-sub activity from 2003-04 to 2006-07 was approximately $93.8M, which includes PERD, CCTII and A-base. Within the same period, other federal government departments contributed $29.1M and non-government sources, primarily Canadian industry, contributed $75.8M. The leveraging ratio of Government of Canada to non-Government of Canada funding was 1.6:1 for the Transportation S&T Sub-sub Activity.

Table 5 below provides estimates of the percentage of cash and in-kind funding leveraged from funding sources outside the Government of Canada for each of the six programs from 2003-04 to 2006-07. These funding sources include: industry; universities; international institutions; provincial/municipal governments; and NGOs.

Table 5:
Estimates of Funding Leveraged from Non-GoC sources per Transportation S&T Program ($K)
Program Fiscal Years Leveraged Amount Total Cost % Leveraged
CTFCA 2001-02 to 2007-08 32,867 64,795 51
HEE 2001-02 to 2007-08 28,199 66,883 42
CLiMRI 2003-04 to 2006-07 5,619 13,379 42
T&I Transportation 2003-04 to 2007-08 2,567 11,358 23
AFTER 2001-02 to 2006-07 4,327 24,036 18
Particulate Matter 2001-02 to 2006-07 2,229 18,247 12
Total Leverage   75,808 198,698 38%

Source: a) OERD financial records; b) program annual reports; and c) program financial project databases.

Program Efficiency
The second indicator used to assess the cost-effectiveness is the management cost: projects costs ratio. Because the Program activity of ‘management’ has not been clearly defined in any of the Programs, each program has included different activities under this heading.98 This presents a challenge in comparing the efficiency (i.e., delivery cost) of each program.

For most of the programs, the funding of the salary of each Program Leader was not accounted for. The Program Leaders dedicate a significant amount of time and effort to making their

Programs achieve the expected outputs and outcomes. PERD does not provide salary funding as the position is considered to be of a volunteer nature, and the salary comes from the departmental A-base to which the Program Leaders belong to. For example, the AFTER Program, which is led by NRC, does not receive funding for the Program Leader’s salary as it is paid by NRC’s A-base funding. Without the salary information of the Program Leaders, the efficiency of program management can not be accurately presented in this evaluation.

The evaluation evidence indicates that there are similarities in program objectives and R&D activities of the Transportation S&T Sub-sub activity areas; and therefore, there has been some duplication of effort for S&T performers with respect to participating on various committees, submission of proposals, project monitoring and reporting. Similarities between the T&I Transportation Program with the existing PERD transportation programs led to a number of projects that were jointly funded between T&I and the PERD programs. In some cases, the T&I Transportation Program reported the same results and funding for the projects reported by the Particulate Matter, CLiMRI, and AFTER Programs.

Due to the lack of financial information on program management tasks (strategic planning and administration) and the salaries of program leaders, it is difficult to determine the exact costs of administrating each program. The findings by program are presented below.

AFTER
The AFTER Program was found to be effective in maximizing its potential impact by leveraging additional resources through other organizations. Approximately $4.3M (18 percent) of AFTER funding was leveraged from sources outside of the Government of Canada.

The program management related activities of the AFTER Program, such as outreach and administration which included sharing of knowledge generated by AFTER and managing the portfolio of projects, received approximately six percent of the total funding. It was unclear as to how much was allocated to outreach activities since most of the projects included some sort of outreach element; and therefore, the overall level of effort is unknown.

HEE
The HEE Program was effective in leveraging approximately 43 percent of its funding from non Government of Canada resources.

The Program did not track program management expenditures (or budgets) and therefore it was not possible to determine the efficiency of the Program. The HEE Program was made up of two Programs: one funded by PERD and the other funded by CCTII. Both Programs were managed by one program manager and management committee so that duplications could be avoided and efficiency could be increased in terms of strategic planning and reporting.

Particulate Program
The Program was found to be an appropriate and effective means to generate knowledge on emissions, but the Program needs to strengthen the essential linkages between research activities and their use in informed decision-making.

From 2001-02 to 2006-07, the Program leveraged approximately 12 percent of its total funding from non Government of Canada resources. Approximately five percent of the PERD funding was allocated to program management for the same period. However, it was not clearly defined in the Program’s annual reports or action plans as to what tasks are included in management.

T&I Transportation
From 2003-04 to 2007-08, approximately three percent ($205K) of the CCTII total funding for T&I Transportation was budgeted for program management. The Program, however, does not clearly define what constitutes ‘management’ and, unlike the activity areas, it does not define what tasks are included in ‘management’ in its Action Plan 2005.

The evaluation evidence indicates that there was overlap between T&I Transportation, which ended in March 2008, and the PERD Transportation programs. There were a number of similarities between the PERD transportation related Programs and the T&I Transportation Program with respect to technical activities (e.g., similar projects with slight nuances), location on the innovation spectrum, and the same federal players involved in the Programs. This was expected since T&I was based on the PERD platform and was composed of four independent themes that were directly aligned with the existing PERD POLs.

However, a number of projects were jointly funded between the T&I Transportation Program and the PERD programs. In some cases, T&I reported the same results and funding for the projects reported by the Particulate Matter, CLiMRI, and AFTER Programs.

As a result of the similarities in the objectives and R&D activities between PERD and T&I, there has been a duplication of effort for S&T performers with respect to participating on various committees, submission of proposals, project monitoring and reporting. This is partly because there were differences in the areas of planning and reporting requirements as a result of different primary mandates (i.e., energy efficiency for PERD; and GHG reductions for T&I).

CLiMRI
The CLiMRI Program leveraged 42 percent of its funding from non Government of Canada resources from 2002-03 to 2006-07.

The Program did not consistently track full management costs. CLiMRI reported budgeting $38K on “strategic planning” from 2002-03 to 2003-04, but no other type of program management activity was accounted for.

CTFCA
The CTFCA leveraged approximately 51 percent of its seven year funding from non Government of Canada resources. Over the seven years, the Program expended $5M in administration costs, including salaries for an average of $605K per year. The Program was unique compared to other programs in that it was funded by CCAP 2000 and CCTII, and did not receive PERD funding.

Potential for Economic Impacts and GHG Reductions
Three key factors serve as context for the potential economic impacts and the reduction of GHG emissions of RD&D projects. First, GHG reductions and economic impacts can only be achieved to the extent that the technologies and knowledge are ultimately commercialized by industry.

The data indicates that the six programs built knowledge and expertise, and tested and showcased technologies, but they did not commercialise the RD&D results. The programs rely on publications and partnerships with the automotive supply chain for technology transfer. Transfer of technology can be direct or indirect depending on where the project is in the technology production /manufacturing chain. With a few exceptions, each project has an industry partner to facilitate the transfer of technology.

Additionally, it must also be recognized that even when the programs are successful in developing new technology, introducing change requires substantial time and resources. For example, once a vehicle change has been designed and tested, the process of changing the manufacturing and assembly operations begins. Even where proven technologies exist, the lead time for introducing them is on the order of 5-10 years and involves hundreds of millions of dollars of investment. The long lifetime of some vehicle models, approximately 15 years, means that the time to replace the current fleet is decades.

Finally, the nature of some of the results (i.e., technical capacity and product development) can be difficult to measure. Difficulties linking results to specific projects – combined with third party delivery, makes attribution of economic and GHG impacts difficult.

AFTER
While most of the AFTER projects have not had any economic impacts nor are they close to commercialization, two projects “Development of a Particulate Matter (PM) Sensor for Diesel Engines (Smoke Sensor)” and “Laser-Induced Incandescence (LII)” have demonstrated the economic potential of the AFTER Program.

The objective of the Smoke Sensor Project was to develop a commercially viable sensor to allow the exhaust gas recirculation (EGR) to control or adjust the NOx emissions and particulate matter and ensure efficient burn in diesel engines. To date, the project has developed a fully functional prototype that has received a US patent and is currently being tested by an external engine research laboratory. Once the sensor has been validated, it is anticipated that the prototype will move into the demonstration phase to accelerate adoption of the technology. When the sensor is closer to commercialization, market penetration could be estimated; however, currently adoption is too far in the future to quantify this potential.

The second project focused on the development of LII technology, which is an instrument that assists in the detection of particulate matter released by the transportation sector. This technology was developed in collaboration among various government departments, the PERD Particulate Program and private sector partners. NRC, with PERD funding, took the lead in developing and patenting the LII, making it available commercially through Artium Technologies, the licensee of NRC’s Laser-Induced Incandescence technology.99

The high costs of the LII system continue to deter many buyers and limit its commercialization. The LII system costs $100K US, while the laser alone costs $20K US. Consequently, only ten LII systems have been sold worldwide. This technology has been utilized across communities and industries, such as engine manufacturers, carbon black manufacturers and monitoring studies. Further R&D on LII is on-going and is intended to enhance the capabilities of the LII instrument to detect lower concentrations of particulate matter emitted from cleaner transportation sources.100

CTFCA While it is too early to demonstrate significant economic impacts from hydrogen and fuel cell technologies, some impacts have been realized from demonstration work conducted during the life span of the CTFCA. A case study of the B.C. Hydrogen Highway (BCHH), which received approximately $7.1M (53.8 percent) of its $13.2M total budget from the CTFCA, identified that economic benefits were obtained. The CTFCA funding enabled a company to test vehicle fuel systems that led to product improvements and $3.5M in sales of modular hydrogen stations since 2005. The company indicated that estimated sales for 2008 and 2009 could total $15M.101

With respect to the potential for reducing GHGs, the CTFCA did not track GHG emissions for the BCHH due to its focus on developing hydrogen fuelling stations for vehicle demonstrations conducted by the Vancouver Fuel Cell Vehicle Program (VFCVP). The VFCVP concentrated on fuel cell vehicle on-road demonstration and, based on Technology Early Action Measure’s (TEAM’s) calculations using its System of Measurement and Reporting for Technologies (SMART), estimated GHG savings from the VFCVP as 0.11t CO2 eq per 1,000 km traveled. Based on data provided by Hydrogen and Fuels Cell Canada (H2FCC) indicating total vehicle kilometres traveled as 168,114 km, the estimated GHG savings from the VFCVP are 18,500 metric tonnes CO2 eq.

The reduction in GHGs that has occurred as result of the Hydrogen Village has not been quantified.

HEE
A case study on “Aluminum Lined Project – 700 Bar” produced evidence of economic impact from the development of a high-pressure, light-weight storage cylinder to store pressurized hydrogen, used for transportation vehicles. This technology has resulted in sales of approximately 200 cylinders and has helped support employment at the company. NRCan provided $450K (a third of the total project budget) and leveraged three times the funding. The cylinders will be used in the 20 new hydrogen buses ordered by BC Transit for the 2010 Winter Olympics.

Particulate Program
There was no reported evidence of economic impact.

T&I Transportation
The case study on the Reduction of Aerodynamic Drag102 showed some potential for an economic impact. This project was designed to test market ready components that can be added to trucks to reduce drag and improve fuel efficiency and determine which combinations of components result in the greatest drag reduction.

This project had a budget of $100K from CCTII, approximately $60K in A-base funds from NRC, and approximately $50K in in-kind contributions from industry. In order to take advantage of the existing infrastructure and expertise, the project was managed and implemented by NRC with the involvement of seven trucking companies. As a result of this project, to date, 110 truck trailers have adopted some of the components, which will result in approximately $0.8M in savings over five years. Overall, the project will also result in savings of 800,000 litres of fuel and 2,160 metric tonnes CO2 equivalent over the same five-year time period.103

CLiMRI
The Multi Material Lightweight Vehicle Body Architecture (mmLiVBA) project focused on developing light-weight components for automobiles, including ultra-high strength steels for the structure surrounding the passenger compartment and magnesium for front-end structures of automobiles. Techniques and materials developed have allowed a weight reduction of 50 percent for the passenger safety cage. If successfully implemented, the development of magnesium front-end automobile structures could lead to a five to seven percent reduction in fuel consumption. This new technology may be implemented in a 2012 car model, and by 2014, it is anticipated that 38,875 cars could have this technology which would result in a cost savings for consumers of $2.65M and that the new materials would save 6,094 metric tonnes of carbon dioxide equivalents104(CO2 eq), when combining reductions of releases of the vehicles and the refineries.105

Within ten years after launch, with the assumption that the technology will be used on three car models, the economic benefits could potentially reach approximately $10M for Canadian automobile owners, in addition to the environmental benefits.106


98 For example:

  • The PM Program had a PERD budget of approximately five percent ($52K per year) for ‘management’ for the fiscal years 2001-02 to 2007-08. However, unlike the other program theme areas, the PM action plan does not define what tasks ‘management’ include and therefore it remains unclear what it represents.
  • The AFTER Program had a budget of six percent ($126K per year) for ‘management and outreach’ for the fiscal years 2005-06 and 2006-07. This included sharing knowledge generated by AFTER and managing the portfolio of projects. The AFTER Program did not budget money for program management between 2001-02 and 2004-05.
  • CLiMRI had a budget of $38K for strategic planning activities in 2002-03 and 2003-04. It did not, however, budget money for program management in any of the years covered by the evaluation (2001-02 to 2007-08).
  • T&I Transportation had an annual budget of three percent ($41K per year) for administration during all the reporting years (2003-04 to 2007-08). However, this did not include a budget for program management; rather it covered the salary for an administrative assistant.

99AFTER Program Annual Reports 2001-02 to 2004-05.
100AFTER Program Annual Reports 2001-02 to 2004-05. SED, NRCan Transportation S&T Case Studies (July 8, 2008).
101 SED, NRCan Transportation S&T Case Studies (July 8, 2008).
102T&I Transportation Program. Annual Report 2005-06.
103 SED, NRCan, Transportation S&T Case Studies (July 8, 2008).
104 Wikipedia defines carbon dioxide equivalency as a quantity that describes, for a given mixture and amount of greenhouse gas, the amount of CO2 that would have the same global warming potential (GWP), when measured over a specified timescale (generally, 100 years). Carbon dioxide equivalency thus reflects the time-integrated radiative forcing, rather than the instantaneous value described by CO2e.
105 SED, NRCan Transportation S&T Case Studies (July 8, 2008).
106 SED, NRCan Transportation S&T Case Studies (July 8, 2008).

4.3.2 To what extent are the performance monitoring mechanisms effective in providing information on Program/project effectiveness and whether adjustments need to made to improve efficiency and effectiveness?

This section assesses the extent to which the performance monitoring mechanisms of the six programs were effective in providing information on program or project effectiveness. The purpose of developing program-based performance measurement frameworks is to identify what needs to be measured in order to have key data available as input to management decision-making and for communicating results.

All six of the programs were found to have performance reporting mechanisms, usually involving written reports and meetings to share information and, in one case, a Website which is no longer accessible was used. For all six programs, the majority of information reported was focussed on projects, with the exception of financial information. Financial information generally varied within and across the six programs with respect to quality and availability; and expenditure data was seldom reported.

The six programs as a whole did not report on their performance frameworks. Annual reports (one program did not produce annual reports for two years) tended to convey information regarding the technical achievements of individual projects and often did not make the linkages to the desired outcomes. This made it difficult for the reader to use the annual reports to understand progress towards achieving program objectives. This was not the case for project reports produced by private sector project proponents. These reports contained readily understandable information on the results that projects are intended to achieve, the significance of the results in terms of the program’s objective, and the resources involved. Interviewees reported that although meetings were effective in sharing information, program documentation such as annual reports were not always effective in communicating results.

The programs’ performance frameworks themselves were found to have two issues related to performance reporting. First, the performance indicators were not used to report performance information. Secondly, the frameworks themselves either had too many indicators or gaps in identification of indicators (e.g., that failed to bridge the gap between research outputs and program outcomes; or that did not identify how the results of the program would be provided to key stakeholders).

The programs’ annual reports varied in quality and consistency and share the issues described in this section. In the case of at least two programs, interviewees described the benefits of results-based management (RBM), such as helping researchers focus on results. Some felt that RBM, although implemented as a process, was not used effectively to monitor the progress of deliverables.

The details of performance measurement findings for each of the programs are outlined below.

AFTER
Findings:

  • The performance reports describe the technical achievements of individual research projects as opposed to the effectiveness of the Program in progressing towards its outcomes. As a result, the information in these reports is of limited value to track the effectiveness of the Program.
  • The availability and quality of financial data was inconsistent.

Most of the annual report information focuses on describing project level activities, research or experimental findings, to some extent outputs, and the scientific significance of research findings. The annual reports do not: a) address any of the project’s three performance indicators; b) help the reader understand the status of the project with respect to achieving its goal; and c) help the reader understand how this output will contribute to its immediate outcome.

A further challenge in assessing the Program’s performance is that there is a large gap between research outputs and the outcomes described in the framework. There is no information on the reaction/awareness/understanding of specific client and beneficiary groups that must receive understand and accept the research results and then act to turn the outputs into the identified outcomes. One grouping is made up of the NRCan and Environment Canada policy and regulatory groups that set fuel and emissions standards and regulations. This group must accept and use the research results if the Program’s work is to contribute to standards and regulations. The second group involves those firms that must produce advanced fuels and manufacture engine and after treatment technologies that, when purchased and used in the transportation sectors, would result in reduced GHG and pollution related emissions.

Reaching these groups is not a forgone conclusion. There may be differences in timing among research and policy groups, e.g., the ‘window’ for research to develop data as input to policy may not be compatible with the time needed to obtain reliable research results. In addition, the research results themselves must be credible, relevant to the needs of the target audience and presented in a user-friendly form that the users can understand. This type of information has not been measured by the Program.

Despite these challenges, interviewees generally agreed that adopting results-based management (RBM) has helped to focus researchers on producing deliverables and reporting on outputs and their significance. However, one member of the Management Committee stated that reporting was extremely difficult to understand. Another interviewee commented that reporting allowed the Management Committee to make a decision to re-allocate funds when a project reported that it was not producing results.

CLiMRI
Findings:

  • CLiMRI performance reporting is based on the technical achievements of individual research projects, and not on the desired outcomes of the Program. As a result, the information is of limited use for describing the performance of the Program; for informing decisions on how to improve the delivery of the Program; and for communicating results to client groups.
  • The availability and quality of financial data was inconsistent.

The evaluation team noted the following issues with CLiMRI’s performance reporting:

  • The annual reports contain discussion on the key achievements of individual projects and provide a description of activities and outputs. However, the linkages of projects and outputs to achieving the desired outcomes are not usually clear.
  • There is a lack of information with regard to the Program’s final outcomes. None of the annual reports reported on the indicators and targets for the final outcomes even though the final data collection period for all the final outcomes was the end of 2006.
  • The achievements that are described are anecdotal and technically-based. The reports do not use the established indicators and targets to report on performance. The indicators themselves are not uniformly strong quality indicators--some describe sources of data, while others are more obscure. As a result, there has been limited reporting on outcomes.
  • The quality and reliability of information in the annual reports is inconsistent:
    • The annual reports do not contain information on actual project expenditures.
    • In some cases, the same project results in successive annual reports were repeated for several years. For example, a project from 2003-04, reported the same information in the annual report of 2006-07 as it did in 2003-04.
    • Some of the projects were missing project numbers and/or had the same number for two different projects.
    • Reporting on some projects appeared to be incomplete. Some projects were reported on only once with no updates in subsequent years. It was not possible to determine whether these projects were successfully completed or terminated.
  • Most projects have established tasks and deliverables and make general statements on whether the project is on track or not. However, the information is focused on technical aspects and it is often unclear when a project will be completed and what its end result is intended to be.

Some interviewees felt that results-based management (RBM) was implemented as a process, but is not used effectively to monitor the progress of deliverables. Although the Program’s Industry Steering Committee members meet once a year for two days to review progress and to make funding decisions and project leaders submit quarterly reports, several interviewees stated that the monitoring of progress was not as effective as it should be.

CTFCA
Findings: The performance reporting could have been strengthened by reporting on the performance framework, including reporting on established targets, timelines, and project progress. The Program did not produce annual reports for two years, but did provide updates through its website and meeting forums.

The CTFCA had an RMAF and produced annual reports from 2001 to 2005 and a final report in 2008. The Program did not produce annual reports from 2005-06 to 2007-08. The annual reports did not report on the performance framework, but did describe a series of outputs and outcomes with future outlooks and key issues. Interview data indicates that stakeholders would have liked to have “more performance metrics to match the activities to the objectives in the reports”.107 The annual reports did not contain information on all projects, nor present financial data.

The Program was effective in providing updates to its stakeholders through general meetings and working groups. For the years that the CTFCA did not produce annual reports (2005-06, 2006-07), the Program continued to update stakeholders at the Core Committee and Working Group meetings. Interviewees indicated that these meetings were very helpful in providing updates on the hydrogen scene within Canada, and established a forum that enabled the development of research ideas and collaboration on hydrogen and fuel cells related projects.108

CTFCA program management received annual (sometimes semi-annual) reports from project leaders. Annual project reports were required prior to the release of hold-backs.

Interviewees reported that a website did exist that reportedly provided useful information on the Program and some project updates; however, it, is no longer accessible.

HEE
Findings:

  • The performance reporting has been somewhat effective for communicating the project and program results. It could be strengthened, however, by incorporating targets and timelines for each indicator as well as baseline information.
  • The availability and quality of financial data was inconsistent.

Interviewees generally felt that the reporting regime worked well for the Program and that the information on program/project results was made available to the Program Committee. Interviewees felt it helped focus projects on Program objectives and results.

The performance measurement strategy lacked clear targets, timelines, and baseline data which made it difficult to understand the context of the reported performance. For example, one indicator for proton exchange membrane (PEM) fuel cells catalysts is “reduced cost while maintaining performance relative to present day design.” In this case, the Program did not define ‘present cost’ and ‘performance’, and, as a result, it is difficult to determine whether and to what extent costs have been reduced and performance maintained. Several interviewees noted that the hydrogen sector created unrealistic expectations. Appropriate goals, targets and deadlines by all stakeholders could address this situation by establishing a context of what is envisioned and by when it might be achieved.

Particulate Program
Findings:

  • The performance reporting is based on the technical achievements of individual research projects, and not on the desired outcomes of the Program. As a result, the information is of limited use for describing the performance of the Program; for informing decisions on how to improve the delivery of the Program; and, for communicating results to policy and regulatory client groups.
  • The availability and quality of financial data was inconsistent.

The Program’s performance measurement framework identified 76 performance indicators: seven indicators for the intermediate/final outcomes; and 69 indicators for the outputs/immediate outcomes. This is the primary means the Program tracks the degree to which it is achieving its goals and the annual reports are used to communicate these successes.

The framework has been of limited value in providing information on program/project effectiveness partly because the information provided in the annual reports does not focus on the Program’s performance indicators. The annual reports provide a description of activities and to some degree outputs, as well as the scientific significance and/or potential of these activities; however, it is not clear how they relate to the desired outcomes.

Five out of seven interviewees identified synthesising and communicating the research results as a weakness of the Program. These interviewees stated that although the Program and project reports assure the Management Committee that funding is spent as planned, they are not effective in describing the results of the Program. The other two interviewees indicated that the reports and the Program technical meetings (presentations) were useful for researchers to share results with each other. One interviewee did not think that the results were shared externally at all. All interviewees agreed that researchers were generally left to communicate their findings on their own. There were no dedicated resources to do this.

In addition to reviewing the annual reports, the evaluation team conducted a detailed review of five projects. None of the project level progress reports contained information on actual expenditures; rather they presented the budget from the Program’s action plan.

In terms of results, most projects have established tasks and deliverables and make general statements on whether the project is proceeding as planned. However, the information in the reports is focused on technical aspects (i.e., datasets; technical reports; methods for detection and analysis) and it is often unclear when a project will be completed and what its end result is intended to be.

T&I Transportation
Findings:

  • The performance reporting is based on the technical achievements of individual research projects, and not on the desired outcomes of the Program. As a result, the information is of limited use for describing the performance of the Program; for informing decisions on how to improve the delivery of the Program; and, for communicating results to client groups.
  • The availability and quality of financial data is inconsistent.

The Program’s performance measurement strategy includes a number of positive elements, such as linking projects to outcomes; however, the evaluation team noted the following issues regarding implementation of the strategy:

  • There is a lack of information regarding the Program’s final outcomes. None of the annual reports reported on the indicators and targets for the final outcomes even though the final data collection period for all the final outcomes was the end of the T&I Initiative (2007-08).
  • The annual reports contain discussion on the key achievements (including outputs) and the significance of each project related to the outcomes. However, these achievements are anecdotal and technically based. The reports do not use the established indicators and targets to report on performance.

Financial Reporting
There were several issues with the Programs’ finances that limited an exact depiction of the expenditures on the S&T Transportation Sub-sub activity. The issues include: a) data inconsistencies; b) data labelling; c) actual expenditures versus budgets; d) double counting; and e) sub-contracting.

OERD tracks PERD and CCTII funding which is distributed to the S&T Transportation programs and to the other government departments involved. OERD does not track program expenditures or how much funding these programs receive from other sources. It is the responsibility of the individual programs to track their funding from various sources, such as OERD, A-base, OGD, industry, other levels of government and NGOs, and to provide this information in their annual reports. CTFCA was managed separately by NRCan’s CETC (CANMET Energy Technology Centre) and it tracked funding received from CCTII and CCAP.

  1. Data Inconsistencies
    The funding data provided in the programs’ annual reports were not consistent with the overall funding levels per program reported by OERD and CTFCA. In addition, the funding data reported in the programs’ annual reports were not always consistent with the funding reported in the programs’ project financial databases. As a result, it was difficult to distinguish which information was accurate. Therefore, the evaluation used OERD financial data to show the PERD and CCTII funding that was distributed to the programs and OGDs. The evaluation also used the program annual reports and project financial databases to show the funding breakdown by activity area and the funding received from all sources, including industry, other levels of government, NGOs and international agencies.

    For the CTFCA, the evaluation used the Program’s financial project database because the annual reports did not contain financial information.

  2. Actual expenditures vs. Budget
    Expenditure information was usually not reported by the programs in their annual reports and project databases. The data provided in the annual reports and in the project databases came from planned project budgets, not actual expenditures. OERD financial information consisted of actual expenditures that it distributed to the programs. For the CTFCA, the Program’s financial records were based on budget commitments from contribution agreements, and the O&M and salary information were based on expenditures.
  3. Double Counting
    There were several double counting entries among the S&T Transportation Sub-Sub Activity areas. For instance, projects funded by the T&I Program reported the same amount and source of funds as the projects funded by the PERD Program and the same results were reported by both Programs. As a consequence, it was difficult to determine to which Program the outcomes should be attributed since both reported receiving the same amount of funding from the same sources. This type of reporting also impeded calculation of leverage and cost effectiveness. Although some programs do occasionally address this situation by including cross-references, such is not a consistent practice.
  4. Labelling
    Funding sources were sometimes not clearly identified and were lumped into one category as “Other” or “Other Government”. Some programs used this category for federal government programs such as TEAM. Furthermore, many of the tables in the annual reports did not clearly explain which departments formed the OGD funding category. Without the identification of which organization contributed how much, a true partnership picture could not be presented.
  5. Sub-Contracting
    Some departments received funding through sub-contracts from OGDs which made it difficult to track the amount of funding OGD recipients received from within a particular program. For instance, EC received funding through sub-contracts with the AFTER Program, making it difficult to track the amount of funding EC received for a particular research activity.

107 Interview data.
108 Interview data.

4.4 Lessons Learned

The following are some of the lessons learned relating to the planning and implementation of the RD&D projects:

  • The development and encouragement of partnerships and collaboration with departments, and external partners such as industry and universities were essential for successful projects. Collaboration facilitates knowledge and technology transfer, and provides the opportunity for sharing resources.
  • The PERD approach was perceived as providing a good exchange of information among researchers (for example, on methodology) and research findings for projects that are complementary in nature. The multidisciplinary approach to projects is a PERD strength because it brings together a variety of backgrounds and expertise for the research projects. For example, the PM Program involves researchers from HC, EC and the NRC. Some of the research findings on particulate matter measurements and characterization provide pertinent information for other studies currently being conducted by other transportation programs.

    However, interviewees suggested that there was an overall lack of integration across projects among the Transportation S&T programs. This suggests that a mechanism to share project findings amongst project ‘teams’ would be a welcome addition.

  • Developing a sound understanding of the automotive industry and the related industries is a key to developing a commercially viable technology. This has taken a great deal of effort (in terms of time and resources).
  • Speed is paramount in the automotive industry as industry needs to bring products to market rapidly to maintain a competitive edge. The faster that R&D can be completed, the more likely it is that industry will participate, and the sooner a product will get to market.
  • It is important to give full consideration to manufacturing and testing equipment. Technology cannot be developed and verified if suitable prototype manufacturing and testing capabilities are not in place.
  • RD&D projects are high risk and delays have to be considered to be part of the normal R&D planning process. Flexibility in adjusting contract tasks, funding and timing is necessary.
  • It is important to use a project management approach with clearly defined tasks and deliverables.
  • Multiple approaches and contingency plans should be considered in new technology development. Contingency approaches should be available. Concurrent development of approaches may be necessary to progress in a timely fashion.
  • It is important to give careful consideration to the selection of possible participating partners and to estimate their capacity.
  • It is important to identify and to keep focussed on the end use of the research.
  • Support from key partners for supplying materials can be difficult if only a few (monopolistic) suppliers exist, and market conditions do not motivate cooperation.
  • It is important to thoroughly plan (including contingencies) and to obtain involvement/commitment from key suppliers at the planning stage.
  • It is important to periodically re-evaluate market trends and whether the R&D project still makes sense in light of technological and market trends.
  • Outreach and communication activities contributed significantly to projects that were successful.
  • NRCan's establishment of good relations with project proponents has been important for the successes realized to date.
  • t is important that the users of the RD&D results are involved at all points of the process.

Annex A: Summary of NRCan’s Transportation S&T Programs

Table A-1:
Summary of NRCan’s Transportation Science and Technology Programs
PERD Programs Name Main Research and Activity Themes
 

1. Support The Development Of Technological And Other Measures To Control And Reduce Emissions Of Particulate Matter (The Particulates) POL 2.1.1

Funding for last 5 years (2002-03 to 2006-07):
NRCan109: $5.8M
OGD: $10.2M
Non-GoC110: $2.2M

Number of cycles
Cycle 1: 2000-01 to 2004-05
Cycle II: 2005-06 to 2008-09

Lead
Toxic Emissions Research and Field Studies,
Environment Canada

This Program aims to provide knowledge and tools that will support the development of technological and other measures to control and reduce emissions of particulate matter and its precursors from transportation sources. More specifically, the goal is to strengthen the scientific basis for policy and regulatory decisions affecting transportation-related emissions of particulate matter and its precursors.The Program conducts the following activities:

  1. Emissions Characterization – Develop technology and methods to measure and describe the emissions of particles, precursors and tracer compounds associated with transportation-related sources;
  2. Ambient PM Characterization – Develop technology and methods to measure and describe the atmospheric presence, transformation and fate of transportation-related particles, precursors and tracer compounds;
  3. Air Quality Modelling – Develop and apply atmospheric modeling techniques and systems for analysis of particles, precursors and tracer compounds associated with transportation-related sources;
  4. Health and Environmental Effects – Develop measurement and analysis techniques, systems and studies to identify acute and chronic effects on humans and the environment (air quality) of particles and other pollutants associated with transportation-related sources; and
  5. Outreach and POL Management – Effective, accountable and transparent POL management, and communication of results within the POL and beyond to partners, the science, policy and regulatory communities and other interested parties.
 

2. Advanced Fuels And Transportation Emissions (AFTER) POL 2.1.2

Funding for last 5 years (2002-03 to 2006-07):
NRCan111: $12.1M
OGD: $7.6M
Non-GoC112: $4.3M

Number of cycles
Cycle 1: 2000-01 to 2004-05
Cycle 2: 2005-06 to 2008-09

Lead
Combustion Research, ICPET
National Research Council

The POL is expected to lead to the development of new and innovative technology using new products and new processes that are marketable. Specifically,  the AFTER’s contribution is expected to be manifested in new fuel and engine technologies designed to reduce emissions and produce a cleaner environment, on top of creating new markets and increasing hydrocarbon sales and oil sands crudes. The Program conducts the following activities:

  1. Fuels Composition and Performance – Investigate the impact on regulated and unregulated emissions due to fuel chemistry for a range of engines and fuels;
  2. Novel and Advanced ICE Technologies – Research on advanced engine technologies for light duty and medium/heavy duty applications;
  3. Engine Hardware and Exhaust Aftertreatment – Develop novel sensors for on-vehicle emissions detection, measurement and characterization of on road and off road vehicle emissions, new laser optical characterization techniques and SCR aftertreatment approaches to reduce emissions;
  4. Health and Environmental Effects – Identify and mitigate potential adverse health and/or environmental effects of innovative emissions reduction technologies; and
Outreach and Administration – Share knowledge gained by AFTER and Leveraged activities; manage the portfolio of projects.
 

3. Canadian Light Weight Materials
Research Initiative (CLIMRI), POL 2.2.1

Funding for last 5 years (2002-03 to 2006-07):
NRCan113: $7.2M
OGD: $0.6M
Non-GoC114: $5.6M

Number of cycles
Cycle 1: 1999-00 to 2002-03
Cycle 2: 2003-04 to 2006-07

Lead
Materials Technology Laboratory,
Canadian Centre for Mineral and Energy Technology,
Natural Resources Canada

The Canadian Light Weight Materials Research Initiative intends to develop and implement lightweight and high-strength materials in transportation applications for the purposes of reducing greenhouse gas emissions through improved vehicle efficiency through improving the competitive performance of the Canadian primary metals, automotive, truck, rail car and aircraft manufacturing industries and their associated parts suppliers. The Program conducts the following activities:

  1. Consumer vehicles: Improve Structural Systems (Body and Frame Components)-
    This activity focuses on materials and processing technologies that are particularly promising for vehicle weight reduction for body-in-white structures, including sheet metal components such as external panels, roofs, hoods and deck lids, and for structural frame components such as space frames, A, B and C pillars, engine cradles, radiator supports and undercarriage frames.
  2. Consumer vehicles: Improve Manufacturing Technologies (Power train, thermal management, braking systems
    This activity focuses on the special problems associated with reducing the weight of components used in the power train, thermal management (heat exchangers) and braking systems. These components have special engineering performance requirements, and all tend to face a requirement of retaining good mechanical properties when hot.
  3. Mass Transit Vehicles
    Improve component and vehicle systems in buses, such auxiliary power-generating systems for buses.
  4. Transportation Fuel Systems (POL 2.1.3) The objective of POL 2.13 was to develop and demonstrate the feasibility of producing transportation fuels from sources that are renewable and economically and environmentally advantageous. It was merged into the Bio-based Energy Systems and Technologies POL which is not a component of this evaluation. POL 2.1.3 was not included in this evaluation.
 

5. Optimization of the energy efficiency of Transportation Systems (POL 2.2.4)

The objective of POL 2.2.4 was to improve the energy efficiency of transportation systems and thereby reduce energy consumption, and associated greenhouse gases (GHGs). POL 2.2.4 has been cancelled thus and was not included in this evaluation.

T&I Programs

6. Advanced End Use Efficiency: Transportation Program

Funding for last 5 years (2003-04 to 2006-07):
NRCan115: $6M
OGD: $2.8M
Non-GoC116: $2.6M

Number of cycles
Cycle 1: 2003-04 to 2007-08

Lead
Toxic Emissions Research and Field Studies,
Environment Canada

The objective of the T&I Advanced End Use Efficiency- Transportation Program is to advance the development and implementation of promising new technologies to achieve long-term mitigation of transportation’s contribution to climate change thereby strengthening Canada’s technology capacity for a more efficient transportation system. This objective is to be achieved through improvements to mainstream vehicle technologies such as internal combustion engines and transmissions.

The Program conducts the following activities:

  1. Vehicle Materials and Design Efficiencies
    This theme focuses on the efficiency of vehicles and in vehicle design improvements and the materials used to make them. Vehicle weight reduction and changes in vehicle design can make a substantial contribution to increasing vehicle fuel efficiency.
  2. Advanced, Efficient Powertrain technologies
    Vehicle-based efficiencies may also be gained by use of advanced combustion strategies and powertrain designs and by use of advanced low-carbon fuels and fuels from renewable and unconventional sources. Research is based on advanced combustion strategies and powertrain designs that can result in dramatically reduced pollutant emissions and fuel consumption for a range of vehicle applications including auxiliary systems.
  3. Intelligent and Efficient Transportation Systems-
    Research is based on system approaches that allow vehicles to operate in a more efficient manner and will compound the benefits of vehicle level improvements. It will focus on priority challenges of system-level efficiencies such as urban congestion, urban goods movement, cross-border congestion, intermodal freight, integration of modes, and shifting of traffic to less energy intensive modes.
  4. Support to Policy Development and Integration-
    Enhance the knowledge base to develop appropriate standards and codes for market use.
PERD, T&I and CETC combined program

7. Hydrogen Energy Economy (HEE) POL 2.2.5, and Technology And Innovation Initiative (T&I) Hydrogen Economy, and, CETC Hydrogen Fuel Cell & Electric Vehicle R&D

Funding for last 5 years (2003-04 to 2006-07):
NRCan117: $29.9M
OGD: $7.3M
Non-GoC118: $28.2M

Number of cycles Cycle 1: 2003-04 to 2007-08

Lead
Technology Programs
Hydrogen, Fuel Cells and Transportation Energy (HyFATE)
CANMET Energy Technology Centre – Ottawa,
Canadian Centre for Mineral and Energy Technology,
Natural Resources Canada

POL 2.2.5, Hydrogen Energy Economy (HEE), was developed in 2003 by joining two PERD POLs, POL 2.2.2- the Fuel Cells, Electric and Hybrid Vehicles Initiative and POL 2.2.3- Hydrogen Initiative due to similar strategic intent, direction and objective. POL 2.2.5 was managed by CETC-HyFATE of CANMET.

Also in 2003, the Climate Change Technology and Innovation Initiative (CCTII) received its TB approval to develop program areas to advance R&D technologies to reduce GHG. As a result of this mandate, one of the program areas developed was the Hydrogen Economy. This program focuses on using hydrogen from renewable sources on applications such as automobiles and stationary power generators, fuel cells and other H2-powered, which are parallel to the strategic objectives of POL 2.2.5. With similar objectives, a joint coordination of PERD POL 2.2.5 and T&I Hydrogen Economy was seen as being important. CETC, having developed expert knowledge in energy technologies and processes was chosen to act as a delivery agent and to lead projects under both programs (PERD and T&I). The involvement of PERD, CCTII and CETC-HyFATE, created the need for an Executive Committee to provide coordination.

The leader of Hydrogen Energy Economy avoids duplication and provides one report on three elements: a) PERD Hydrogen Energy Economy, POL 2.2.5; b) T&I Hydrogen Economy and; c) CETC Hydrogen Fuel Cell & Electric Vehicle R&D.The Program conducts the following activities:

  1. Hydrogen Production Challenges
    • Increase efficiency of hydrogen production equipment (e.g., by developing membranes and systems that can allow electrolyzers to operate at higher temperatures and pressures.
    • Decrease costs (e.g., materials, catalysts, process modifications etc.)
    • Integration of renewable energy into hydrogen production.
  2. Hydrogen Storage Energy Storage Challenges
    • Cost: Low-cost materials and components for hydrogen and energy storage systems are needed, as well as low-cost, high volume manufacturing methods.
    • Weight and Volume: Materials and components are needed that allow compact, lightweight, hydrogen and energy storage systems while enabling vehicles with a greater that 450km range.
    • Efficiency: For solid-state storage systems, R&D is needed to improve hydrogen capacity and reversibility at practical operating temperatures and pressures and within refueling time constraints.
    • Durability: Materials and components are needed that allow hydrogen and energy storage systems with a lifetime of 1500 cycles for automotive applications.
  3. Hydrogen Utilization
    Challenges
    • Cost: Lower cost, lighter bipolar plates, membrane electrode assemblies, and low-cost high-performance membranes and catalysts are required to make fuel cells competitive.
    • Lifetime: To compete against other distributed power generation systems, stationary fuel cells must achieve greater than 40,000 hours lifetime within a temperature range of –35OC to 40OC for widespread commercialization.
    • Stack Performance: Higher temperature membranes are needed for PEM fuel cells to lessen water accumulation and management issues, to improve cell performance and lessen heat management issues caused by low temperature differential available for fuel cell system heat rejection.
  4. Codes, Standards, Policy and Outreach
    • This theme will support R&D in support of codes, standards and safety; R&D that will provide input into policy and decision-making, as well as outreach activities
TEAM

8. TEAM Transportation Components

Funding for last 5 years
(2003-04 to 2007-08)
TEAM: $10.3 M
OGD: $5.9 M
Non-GoC119: $37.7M

Number of cycles
Cycle 1: 1998-99 to 2000-01
Cycle 2: 2001-02 to 2002-03
Cycle 3: 2003-04 to 2007-08

Lead
TEAM Operations Office (a separate entity until Aug 2007, now integrated with the Office of Energy R&D)

The overall TEAM mission is to identify, develop and support technology late stage development and demonstration projects and technology transfer opportunities in support of early action to reduce GHG emissions, domestically and internationally, while sustaining economic and social development. The TEAM Transportation components are:

  1. Advanced End-Use Efficiency Technology
    • Transportation vehicles, modes and systems
  2. Hydrogen Economy
    • Stationary and transport fuel cell & hybrid applications, associated enabling technologies (electronic inverters), Hydrogen infrastructure & refuelling (H2 extraction and conditioning), as identified by the Early Adopters Initiative
CETC and T&I Program

9. Canadian Transportation Fuel Cell Alliance (CTFCA)

Funding for last 5 years (2002-03 to 2006-07):
NRCan120:$31.4M
OGD: $0.6 M
Non-GoC121: $32.9M

Number of cycles
Cycle 1: 2001-02 to 2005-06
Cycle 2: 2005-06 to 2007-08

Lead
Fuel Cell Infrastructure
Natural Resources Canada

The CTFCA has two components, which are: to demonstrate the greenhouse gas reductions and evaluate different fuelling routs for fuel cell vehicles; and to develop the necessary supporting framework for the fuelling infrastructure, including technical standards, codes, training, certification and safety. These activities are essential to ensure that fuel cell vehicles become a viable commercial option.

  1. Fuelling demonstration for light duty vehicles and medium duty/heavy duty vehicles (transit buses) and associated analyses
    • Light-Duty Vehicle Fuelling Demonstrations
    • Medium/Heavy duty Fuelling Demonstrations
  2. Development of standards and procedures, training and certification of personnel, development and testing of safety systems
    • Standards and Procedures
    • Training and Certification
    • Safety
  Total Funding for last 5 years
(2002-03 to 2006-07)
NRCan: $93.8M
OGD: $29.1M
Non-GoC: $75.8M

109 Includes NRCan A-Base, PERD and CCTII sources of funds.
110 Includes funding from universities, industry, other levels of government-provincial, NGO and international.
111 Refer to footnote 107.
112 Refer to footnote 108.
113 Refer to footnote 107.
114 Refer to footnote 108.
115 Refer to footnote 107.
116 Refer to footnote 108.
117 Refer to footnote 107.
118 Refer to footnote 108.
119 Refer to footnote 108.
120 Refer to footnote 107.
121 Refer to footnote 108.

Overview of Program of Energy Research and Development (PERD)

The Program of Energy Research and Development (PERD) was created in 1974 after the first oil crisis in response to the request from the Organization for Economic Cooperation and Development (OECD) for a coordinated effort to manage energy supply and use between developed nations. PERD is a $58M per year interdepartmental R&D program that directly supports 40 percent of all non-nuclear energy R&D conducted in Canada by the federal and provincial governments, supporting both fundamental and applied R&D. Its close affiliations with federal programs that focus on technology deployment and commercialization ensure that energy R&D is an important part of the federal government’s innovation agenda. The objective of PERD is to provide the continuing S&T necessary for Canada to move towards a sustainable energy future.

The Office of Energy Research and Development (OERD) within the Energy Policy Sector of NRCan has overall management responsibility for PERD. PERD funding for approved R&D projects is provided under Memoranda of Understanding (MOUs) signed between NRCan and the twelve participating federal departments. These MOUs establish a framework for collaboration; outline the scope of activities supported by PERD, the roles and responsibilities of the parties, the mechanism to transfer funds, reporting requirements, publications and intellectual property. Participating departments team-up with R&D delivery agents including federal laboratories; the private sector (e.g., industry, research institutes, companies, consortia and alliances, and individuals); associations; other funding agencies; universities; provincial and municipal governments; research organizations; and international organizations.

PERD is divided into six ‘strategic intents’ as outlined in the 1999 NRCan Energy S&T Companion document. These strategic intents are further broken down into ‘strategic directions’ that provide more detail, and finally, a number of objectives are associated with each strategic direction. These objectives outline what is to be achieved and form the basis for the PERD POLs. A POL, or ‘program at the objective level’, is a set of R&D projects that are all targeted toward the same objective. Beginning in 2000, PERD underwent changes to improve its strategic alignment to departmental, intra-departmental and government-wide priorities, and international commitments resulting in the discontinuation of some POLs and the merging of others.

The Strategic Intent that applies to PERD’s Transportation related programs is:

Strategic Intent 2: Foster cleaner sustainable transportation fuels and systems in order to improve the environment, reduce emissions, including GHGs, and to increase economic activity through development of domestic and export markets.

In May 2007, OERD restructured, replacing the PERD POL concept with a more broadly defined Portfolio model. Eight Portfolios, each comprised of a number of PERD (A-based activities) and grants and contributions activities were created. PERD POLs retained their strategic intents, directions, and objectives as described above, but are now referred to as Programs rather than POLs. Remaining T&I and TEAM activities have been incorporated into the Portfolios, as have the ecoEnergy Technology Initiative activities.

Overview of the Climate Change Technology and Innovation Research and Development Initiative (T&I R&D)

In order for Canada to achieve the goals set out in the Climate Change Action Plan 2000, industry needed R&D and dissemination assistance from the federal government through public/private sector partnerships. The 2002 Climate Change Plan for Canada outlined a broad strategy to address climate change and, following this, Budget 2003 announced $1.7 billion in new climate change funding over five years with a portion dedicated to longer-term technology development.

The 2002 Climate Change Plan for Canada recognized the critical role of innovation in promising technologies by investing $250 M over five years to ‘advance promising GHG technologies through R&D, demonstration and early adoption initiatives to achieve long-term GHG reductions and strengthen Canada’s technology capacity’. Of the $250 M, $115 M over five years was dedicated to the Climate Change Technology and Innovation Research and Development (T&I R&D) Initiative. T&I R&D focuses on the research and development objectives of The Plan to advance knowledge and develop transformative technologies for the purposes of reducing GHG intensity and stimulating the economy in the long-term. TB program authority for T&I R&D was obtained in June 2003 and will terminate in 2007-08.

The 2002 Climate Change Plan for Canada called for a 21 Mt reduction in GHG emissions from transportation-related sources by 2010. This was to be achieved in four areas of the transportation system:

  • increased production and use of low carbon and renewable fuels;
  • increased vehicle fuel efficiency;
  • increased use of public transit and managing growth of vehicles use; and
  • improved performance targets and best practices for all freight transport and enhanced intermodal infrastructure.

These priorities were reiterated in Project Green: Moving Forward on Climate Change-A Plan for Honouring our Kyoto Commitments (2005).

T&I Objectives
The overall objectives of T&I are to:

  • Support long-term research and development, which will translate into cost-effective GHG mitigation technologies;
  • Demonstration of promising new GHG-reducing technologies; and,
  • Support early adoption of hydrogen and fuel cell technologies into the marketplace.

T&I is comprised of a suite of nine programs/expert groups. The programs each have their focus in one of five strategic areas as follows:

  • cleaner fossil fuels
  • advanced end-use efficiency (comprised of 3 expert groups, one of which is Transportation)
  • decentralized energy production, including renewables
  • biotechnology
  • hydrogen economy

The two Strategic Areas that have transportation components are Advanced End-use Efficiency – Transportation, and the Hydrogen Economy.

The objective of the Advanced End-Use Efficiency Strategic Area is: to advance and expand the S&T for highly energy efficient low emission industrial processes, transportation systems, and building and community systems.

The overall objectives of the Transportation R&D program under Advanced End-Use Efficiency are:

  1. to expand the knowledge base by improving our understanding of the role of the transportation sector in climate change; the magnitude as well as the mechanisms involved in this contribution and
  2. to advance the development and implementation of promising new technologies to achieve long-term mitigation of transportation’s contribution to climate change;

…thereby strengthening Canada’s technology capacity for a more efficient transportation system.

The objective of the T&I Hydrogen Economy Strategic Area is expressed in its vision of reducing greenhouse gas emissions and air pollution and increasing energy efficiency and generating wealth by supporting R&D to create a sustainable, competitive hydrogen economy. The R&D portion of T&I Hydrogen Economy is to be accomplished by focusing on cost-sharing partnerships with industry, universities and other research organizations in applied R&D leading to competitive hydrogen systems. Some more fundamental work is also to be conducted to investigate promising new next generation technologies.

Program Mechanisms
T&I R&D has been guided by long-term strategic planning. A Strategic Plan was developed for all T&I R&D in the first eight months of the program that provides a vision to 2025 and that takes into consideration expected energy futures and technology areas that must be addressed to get there. This led to the development of Action Plans that describe activities that could be carried out over the five year duration of the program, and could make progress towards achieving the vision of 2025. Each Action Plan was critically tested by internal and external experts to identify strengths and weaknesses against three primary criteria: relevance, risk, and impact. Once adjustments were made, the Action Plans were used in preparing requests for proposals.

The first ‘full’ year of the T&I Program was considered as being 2004-05, following a year of planning. Project proposals were requested from the federal S&T community and were selected by the Expert Groups according to a minimum of four obligatory criteria, including:

  1. Potential GHG impact;
  2. Partnership and leverage with industry, provinces and academia;
  3. Alignment with the Strategic Plan; and,
  4. Dissemination activities

T&I R&D is coordinated by Expert Groups made up of interdepartmental S&T experts who report to the Interdepartmental Directors General Steering Committee. The Steering Committee, chaired by NRCan, is comprised of nine different departments, three central agencies and TEAM. The Committee approves funding allocations and makes recommendations on the identified strategic areas including mid-course corrections. It monitors approved R&D programs, tracks progress and reviews outputs and outcomes. The Expert Groups consult an External Advisory Group comprised of relevant stakeholders on strategic directions, theme selections, and relevance.

The Directors General Steering Committee and Expert Group are supported by the T&I Management Secretariat (TIMS), responsible for the definition, implementation, and administration of the T&I R&D Program. Periodically, T&I Management and Secretariat and the Directors General steering Committee request additional project-level information before recommending release of funds. All funding transfers between the Office of Energy Research and Development (OERD) at NRCan and the S&T performing organizations are covered by formal Memoranda of Understanding.

Overview of the Canadian Transportation Fuel Cells Alliance Initiative

The Canadian Transportation Fuel Cell Alliance is a $33M federal government initiative that began in 2001 and ended in March 2008. The Program was originally established under Action Plan 2000 as a five year, $23M sunset Program ending in March 2006. The Program was then extended through the CCT&I Initiative with an additional $10M in funding over two years (ending in March 2008).

The CTFCA has two components, which are: to demonstrate the greenhouse gas reductions and evaluate different fuelling routs for fuel cell vehicles; and to develop the necessary supporting framework for the fuelling infrastructure, including technical standards, codes, training, certification and safety. These activities are essential to ensure that fuel cell vehicles become a viable commercial option.

The CTFCA builds on the technology of NRCan’s existing Hydrogen and Fuel Cell R&D Program to take the first steps towards establishing the fuelling infrastructure for fuel cell vehicles. NRCan’s existing Hydrogen and Fuel Cell R&D Program is aimed at developing the technology for hydrogen production, storage, safety and utilization. This existing Program has:

  • reduced the footprint and the capital and operating costs for hydrogen production from water electrolysis:
  • increased hydrogen storage capacity (by volume and weight) of metal hydrides, carbon structures and pressure cylinders;
  • developed materials for the magnetic liquefaction of hydrogen to increase practicality and lower costs;
  • investigated and established hydrogen safety measures and equipment; and,
  • developed fuel cells for stationary and transportation applications.

The CTFCA is initiating the establishment of fuelling infrastructure through a number of co-funded projects that provide specific opportunities for learning and solving technical and economic issues associated with the introduction of fuelling for fuel cell vehicles in Canada.

The Program includes the participation of fuel cells suppliers, fuel providers, the automotive industry, the federal government, and provincial governments.

Overview of the Technology Early Action Measures Initiative

Technology Early Action Measures (TEAM) is an interdepartmental, technology investment Program that was first established in 1998 under the Federal Government's Climate Change Action Plan. The Program operates under the leadership of Natural Resources Canada (NRCan), Environment Canada (EC) and Industry Canada (IC), with the participation of several other federal government departments. It is uniquely positioned to finance late stage development and first demonstration of new technology which is an important area of investment needed to bridge R&D and commercial market implementation for new technologies. This zone is where companies find investment dollars and technical assistance very scarce, at the very time when they are most needed, to enable the move from R&D concept to business reality.

Following Budget 2003 TEAM received additional funding through the Climate Change Technology and Innovation initiative (T&I) and as result adopted the initiative’s five major priority areas for investment. These areas are: (1) cleaner fossil fuels, (2) advanced end-use technology, (3) biotechnology, (4) hydrogen economy, and (5) decentralized energy production.

The scope of this evaluation is includes TEAM’s activities as they relate to transportation S&T, which includes the priority areas (2) advanced end-use technology [for transportation] and (4) hydrogen economy. The transportation related demonstrations account for roughly $27.2M in expenditure between the Program’s inception and the fall 2007, which is equivalent to roughly 20 percent of the total TEAM expenditures.

Mission and Objectives
The overall TEAM mission is to identify, develop and support technology late stage development and demonstration projects and technology transfer opportunities in support of early action to reduce GHG emissions, domestically and internationally, while sustaining economic and social development.

TEAM focuses on accelerating the market implementation of technologies for climate change and economic benefits through the following three objectives:

  • Supporting technology deployment and development by addressing technology deployment projects over the short-term (less than 3 years), at or near market stage, and research and development projects over the medium-term (3 to 5 years), near demonstration stage;
  • Overcoming non-technological barriers to technology deployment and development; and
  • Piloting technology transfer to developing countries and countries in transition

Program Delivery
The TEAM Program is delivered through the Director TEAM Operation Office (TOO) who reports to the TEAM Executive Committee and the ADM Energy Technology and Program Sector. The TOO is responsible for TEAM management, coordination and reporting, marketing, initial screening of proposals, providing recommendations to the TEAM executive Committee, managing funds, calculating GHG reductions, and acting as a interface with stakeholders and clients. The TEAM Executive Committee is responsible for approval of projects and the ADM is responsible for authorizing the transfer of funds.

All TEAM projects are vetted through the 13 step selection and approval process. Funding is allocated based on:

  • eligibility criteria
  • risk factors
  • replication potential
  • leverage partnerships
  • potential environment/health benefits
  • potential socio/economic benefits
  • need for government investment

Overview of Hydrogen Energy Economy Program

The Canadian government has a longstanding involvement in the development of hydrogen and fuel cell technologies that began in 1978 when NRCan signed the International Energy Agency (IEA) Implementing Agreement on Hydrogen. Programs in this field have grown since then as environmental concerns have drawn attention to fuel cells and their ability to produce energy more efficiently while reducing emissions. This section outlines the evolution of hydrogen R&D programs in NRCan for the proposed scope of evaluation (2000-01 to 2006-07).

In September 2000, a triennial plan (2000-2003) had been developed for the Fuel Cells, Electric and Hybrid Vehicles Initiative - POL 2.2.2 and the Hydrogen Initiative - POL 2.2.3. These two POLs were funded under the Program of Energy Research and Development. POL 2.2.2 worked on fuel cells and hybrid and electric vehicles, directing its efforts to R&D for advanced electric vehicle technology. The Hydrogen Initiative POL 2.2.3 worked towards enhancing and supporting the development of cost-competitive hydrogen production, utilization, storage and safety technologies for portable, stationary and transportation applications. Both of these POLs were evaluated in 2003.

Parallel to the two evaluations, in May 2003, the Office of Energy Research and Development (OERD) conducted a study, the Strategy for Hydrogen Energy Economy to document the rationale for merging POLs 2.2.2 and 2.2.3 into POL 2.2.5 - Hydrogen Energy Economy. The study concluded that the two existing POLs were similar in vision, objectives, and activities; both were funding applied R&D in fuel cells and hydrogen energy in partnership with industry and universities. In addition, both POLs were focusing on cost-sharing with industry with the majority of the work conducted by the industry partners, and to a lesser extent universities and government laboratories. Therefore, the two POLs were combined to form the new POL 2.2.5, Hydrogen Energy Economy.

In 2003/04, the Technology and Innovation Initiative (T&I) was launched as a five year initiative, with a budget of $250M. The Initiative was expected to promote and improve innovation in cleaner energy by focusing on cleaner fossil fuels, advanced end-use efficiency technologies, decentralized energy production, (including renewables), biotechnology and the hydrogen economy. With each component, two types of activities were to be undertaken: the development of technologies and the demonstration of prototypes to raise awareness of the technologies and prove their technical feasibility. Of the TII total budget, the hydrogen economy component was given $80M: seven million dollars of this amount was allocated to hydrogen and fuel cell R&D, and was being administered by OERD under a management structure similar to PERD.

Due to the similar nature of the goals of PERD and TII in the hydrogen economy area, it was likely that there was considerable similarity between what would have been eligible for PERD or TII funding. Both PERD and the TII Hydrogen related components promised to expand the knowledge base and the advanced technologies that will mitigate climate change and air pollution through hydrogen and hydrogen-related research and development. For that reason, the PERD and T&I programs were combined into one Hydrogen Energy Economy Program (HEE) that would be delivered by the CANMET Energy Technology Centre HyFATE unit.

Hydrogen Energy Economy Objectives
The HEE is expected to meet the Strategic Intent and Strategic Direction for research and development as defined in NRCan’s Energy S&T framework. The framework defines the Strategic Intent and Strategic Direction for R&D as follows:

  1. Strategic Intent 2 - Foster cleaner sustainable transportation fuels and systems in order to improve the environment, reduce emissions, including GHG, and to increase economic activity through development of domestic and export markets.
    1. Strategic Direction 2.2 – Science and technology to improve energy efficiency reduce emissions and provide economic benefits to Canada from next generation vehicles and systems.
      1. Objective 2 - The development of fuel cell, electric and hybrid vehicle components and their supporting infrastructures.
      2. Objective 3 - The establishment of GHG neutral production of hydrogen and its storage and supporting fuelling infrastructure. The overall strategy for the achievement of this objective is to conduct R&D that will improve the competitiveness of hydrogen production, utilization and storage while ensuring adequate safety to users.

Annex B: Funding Sources by Transportation S&T Sub-sub Activity

This Annex provides the financial information for each program covered by this evaluation. The sources of financial data (not applicable to the CTFCA) include:

  1. OERD financial records contained the overall PERD expenditures for all the Transportation S&T programs and the evaluation used these records as the primary source of information.
  2. Program Annual Reports contained budget information (i.e., not actual expenditures) by theme/activity area for PERD/CCTII funding as well as for contributions (budgeted) from other funding sources. These financial records were used as OERD records do not contain this information.
  3. In most cases, resources provided by non-PERD/CCTII sources did not distinguish between cash and in-kind.
  4. In cases where sources of funding were categorized as “other”, they were reported as industry sources.

1) Hydrogen Energy Economy Program

Table B-1:
Estimates of Hydrogen Energy Economy Funding by Source, 2001-02 to 2007-08 ($K)
Funding Sources Fiscal Year Total % of Total
POL 222 & 223 Hydrogen Energy Economy
(POL 225 and CCTII)
2001-02 2002-03 2003-04 2004-05 2005-06 2006-07 2007-08
GoC PERD 4,321 4,190 4,352 4,302 4,311 4,150 4,325 29,951 33
Other Gov’t funding programs (OGFP) 3,413 0 1,000 2,345 310 302 426 7,796 9
NRCan A-base 532 500 964 973 1,265 1,368 909 6,511 7
CCTII       1,157 1,230 1,096 1,230 4,713 5
OGD     581 810 889 1,065 435 3,780 4
Sub-Total 8,266 4,690 6,897 9,587 8,005 7,981 7,325 52,751 58
Non GoC Industry 4,859 0 6,288 6,072 5,916 4,594 3,678 31,407 35
Universities 0 0 651 1,030 647 1,288 389 4,005 4
Other Levels of Government 0 0 1,031 175 130 286 170 1,792 2
NGO 0 0 0 0 91 0 323 414 0
International 0 0 0 0 0 0 0 0 0
Sub-Total 4,859 0 7,970 7,277 6,784 6,168 4,561 37,619 42
Total 13,125 4,690 14,867 16,864 14,789 14,149 11,885 90,369 100

2) Particulate Matter Program

Table B-2:
Estimates of Particulate Matter Funding by Source, 2001-02 to 2006-07 ($K)
Funding Sources Fiscal Year Total % of Total
2001-02 2002-03 2003-04 2004-05 2005-06 2006-07
GoC OGD 1,635 1,005 1,170 1,240 3,680 3,133 11,863 57
PERD 715 759 679 713 1,429 1,265 5,560 27
CCTII         440 415 855 4
NRCan A-base         50 40 90 0
Sub-Total 2,350 1,764 1,849 1,953 5,599 4,853 18,368 88
Non GoC Industry 95 95 272 252 470 609 1,793 9
International 25 30 90 40 73 82 340 2
Universities 36 10 45 45 50 50 236 1
Other Levels of Government 13       8 8 29 0
Sub-Total 169 135 407 337 601 749 2,398 12
Total 2,519 1,899 2,256 2,290 6,200 5,602 20,766 100

3) T&I Transportation Program

Table B-3:
Estimates of T&I Transportation Funding by Source, 2003-04 to 2007-08 ($K)
Funding Sources Fiscal Year Total % of Total
2003-04 2004-05 2005-06 2006-07 2007-08
GoC CCTII 442 457 1,418 1,690 2,016 6,023 34
OGD   435 899 1,430 1,111 3,875 22
PERD   199 1,040 420 1,018 2,677 15
NRCan A-base   10 43 308 408 769 4
Sub-Total 442 1,101 3,400 3,848 4,553 13,344 76
Non GoC
 
Industry   100 290 815 1,028 2,233 13
International     500 500 500 1,500 9
Universities   0 107 155 135 397 2
Other Levels of Government   25 75   20 120 1
Sub-Total 0 125 972 1,470 1,683 4,250 24
Total 442 1,226 4,372 5,318 6,236 17,594 100

4) Canadian Transportation Fuel Cell

Table B-4:
Estimates of CTFCA Funding Sources, 2001-02 to 2007-08 ($K)
Funding Sources Fiscal Year Total % of Total
2001-02 2002-03 2003-04 2004-05 2005-06 2006-07 2007-08
GoC Action Plan 2000 & CCTII 595 1,294 9,031 9,215 4,593 5,971 1,124 31,823 48
OGD   200 2 363 0     565 1
Sub-Total 595 1,294 9,031 9,215 4,593 5,971 1,124 32,388 49
Non-GoC Industry 63 2,036 7,948 9,266 3,351 3,894 289 26,847 40
Other Levels of Government   50 858 773 571 3,623 402 6,276 9
Universities   14     244   223 481 1
NGO     58 24 44 113 285 525 1
Sub-Total 63 2,050 8,006 9,290 3,639 4,007 797 34,129 51
Total 658 3,345 17,037 18,505 8,232 9,978 1,921 66,516 100

Sources: The financial information presented in this table is approximate because the Program’s financial records were based on budget commitments from contribution agreements. NRCan’s government financial system (GFS) and data from the Program were used for CTFCA’s administrative and salary expenditures.

5)Canadian Lightweight Material Research Initiative

Table B-5:
Estimates of CLiMRI Funding Sources,122 2001-02 to 2006-07 ($K)
Funding Sources Fiscal Year Total % of Total
2001-02 2002-03 2003-04 2004-05 2005-06 2006-07
GoC PERD 831 807 999 993 995 995 5,620 36
NRCan A-base 308 188 281 445 458 973 2,653 17
OGD 370 308 15 23 110 110 936 6
CCTII     60       60 0
Sub-Total 1,509 1,303 1,355 1,461 1,563 2,078 9,269 59
Non- GoC Industry 778 1,407 362 966 863 863 5,239 33
Universities 115 130 155 291 291 291 1,273 8
Sub-Total 893 1,537 517 1,257 1,154 1,154 6,512 41
Total 2,402 2,840 1,872 2,718 2,717 3,232 15,781 100

6) Advance Fuel and Transportation Emissions Reduction Program

Table B-6:
Estimates of AFTER Funding by Source, 2001-02 to 2006-07 ($K)
Funding Sources Fiscal Year Total % of Total
2001-02 2002-03 2003-04 2004-05 2005-06 2006-07
GoC PERD 1,786 1,768 1,682 1,754 2,165 2,135 11,290 41
OGD 940 999 1,267 594 2,585 2,195 8,580 31
NRCan A-base 149 335 975 50 210 210 1,929 7
CCTII         490 295 785 3
Sub-Total 2,875 3,102 3,924 2,398 5,450 4,835 22,584 81
Non- GoC Industry 594 483 462 60 1,585 985 4,169 15
Universities 118 155 114 10 45 45 487 2
International 135       163 172 470 2
Provincial & Municipal Governments         8 40 48 0
Sub-Total 847 638 576 70 1,801 1,242 5,174 19
Total 3,722 3,740 4,500 2,468 7,251 6,077 27,758 100

122 Non-Government of Canada sources include cash and in-kind from industry and university.

Annex C: PERD Funding Contribution per Organization by Program

The following financial information was OERD records for all Transportation S&T Sub-sub Activity areas.

Table C-1:
Estimates of PERD Funding to Recipient Organizations for Particulate Matter, 2001-02 to 2007-08, ($K)
Organization $ %
EC 3,870 55
NRC 1,514 22
HC 1,129 16
DND 202 3
NRCan 193 3
TC 141 2
Total 7,049 100
Table C-2:
Estimates of PERD Funding to Recipient Organizations for AFTER, 2001-02 to 2007-08, ($K)
Organization $ %
NRCan 4,615 34
NRC 4,184 31
EC 1,567 12
HC 1,819 13
TC 1,173 9
DND 225 2
Total 13,583 100
Table C-3:
Estimates of PERD Funding to Recipient Organizations for CLiMRI, 2001-02 to 2007-08, ($K)
Organization $ %
NRCan 5,634 85
NRC 530 8
TC 451 7
Total 6,615 100
Table C-4:
Estimates of PERD Funding to Recipient Organizations for Hydrogen Energy Economy, 2001-02 to 2007-08, ($K)
Organization $ %
NRCan 25,983 87
DND 2,343 8
TC 882 3
EC 469 2
HC 274 1
Total 29,951 100
Table C-5:
Estimates of CCTII Funding to Recipient Organizations for Hydrogen Energy Economy, 2003-04 to 2007-08, ($K)
Organization $ %
NRCan 4,253 90
DND 460 10
Total 4,713 100
Table C-6:
Estimates of CCTII Funding to Recipient Organizations for T&I Transportation, 2003-04 to 2007-08, ($K)
Organization $ %
NRCan 2,004 34
NRC 2,310 39
EC 1,476 25
TC 166 3
Total 5,955 100
Table C-7:
Estimates of CCTII (&CCAP) Funding to Recipient Organizations for CTFCA, 2001-02 to 2007-08, ($K)
Organization $ %
NRCan 4,240 86
NRC 398 8
DND 200 4
TC 35 1
IC 25 1
DFAIT 22 0
Western Economic Diversification 20 0
Total 4,940 100

Annex D: Estimates of Funding Contribution from Organizations to Programs

The following financial information was compiled using annual reports from all Transportation S&T Sub-sub Activity areas.

Table D-1:
Estimates of Funding Contributions from other Organizations to Particulate Matter, 2001-02 to 2006-07 ($K)
Organization $  %
NRCan 6,505 31
EC 7,548 36
NRC 2,191 11
HC 1,970 9
Industry 1,793 9
International 340 2
University 236 1
DND 130 1
Other Levels Governments 29 0
TC 24 0
Total 20,766 100
Table D-3:
Estimates of Funding Contributions from other Organizations to CLiMRI, 2001-02 to 2006-07 ($K)
Organization $ %
NRCan 8,333 53
Industry 5,239 33
University 1,273 8
NRC 558 4
TC 378 2
Total 15,781 100
Table D-2:
Estimates of Funding Contributions from other Organizations to AFTER, 2001-02 to 2006-07 ($K)
Organization $ %
NRCan 14,004 51
Industry 4,169 15
NRC 3,480 13
HC 2,045 7
EC 1,837 7
TC 928 3
University 487 2
Int'l 470 2
DND 150 1
OLG 48 0
Total 27,618 100
Table D-4:
Estimates of Funding Contributions from other Organizations to Hydrogen Energy Economy: PERD Portion, 2003-04 to 07-08 ($K)
Organizations $ %
NRCan 25,645 41
Industry 24,890 39
Other GoC Funding Programs 3,836 6
University 3,033 5
DND 2,818 4
OLG 1,792 3
NGO 414 1
EC 356 1
TC 256 0
Total 63,040 100
Table D-5:
Estimates of Funding Contributions from other Organizations to Hydrogen Energy Economy: CCTII Portion, 2004-05 to 2007-08 ($K)
Organization $ %
NRCan 5,987 63
Industry 1,658 17
University 972 10
OGFP 547 6
DND 350 4
Total 9,514 100
Table D-6: Estimates of Funding Contributions from other Organizations to T&I Transportation, 2003-04 to 2007-08 ($K)
 Organization $  %
NRCan 2,004 33
NRC 2,310 38
EC 1,476 24
TC 166 3
University 69 1
Total 6,024 100
Table D-7:
Estimates of Funding Contributions from other Organizations to CTFCA, 2001-02 to 2007-08, ($K)
Organization $  %
NRCan 33,081 49
Industry 26,847 40
OLG 6,276 9
NGO 525 1
University 481 1
NRC 298 0
DFAIT 2 0
Western Economic Diversification 40 0
Total 67,550 100