Evaluation of the Clean Energy Systems for Industry (CESI) Sub-Activity

Table of Contents

Abbreviations

CAC Criteria air contaminant
CANMET NRCan CANMET Energy Technology Centre (Ottawa, Devon, Varennes)
CCTII Climate Change Technology Innovation Initiative
CCRA Canadian Carbonization Research Association
CESI Clean Energy Systems for Industry
CIPEC Canadian Industry Program for Energy Conservation
COREM Consortium of applied research for mineral processing (Quebec)
DOE Department of Energy (United States)
ecoETI ecoENERGY Technology Initiative
EC Environment Canada
GHG Greenhouse gas
HEIST Highly Energy-efficient Industrial Systems and Technologies
ICPET Institute for Chemical Process and Environmental Technology (NRC)
IRAP Industrial Research Assistance Program
NRC National Research Council Canada
NRCan Natural Resources Canada
OEE Office of Energy Efficiency
OERD Office of Energy Research and Development
PERD Program of Energy Research and Development
R&D Research and Development
S&T Science and Technology
SBDA Science-based departments and agencies
SDTC Sustainable Development Technology Canada
SED Strategic Evaluation Division (NRCan)

EXECUTIVE SUMMARY

Introduction

This report presents the findings of an evaluation conducted on the Clean Energy Science & Technology sub-activity: Clean Energy Systems for Industry Portfolio (CESI sub-activity). The S&T research projects are undertaken in federal laboratories in concert with industry to improve and develop industrial processes and technologies that will reduce industrial energy intensities and lower greenhouse gas (GHG) emissions and criteria air contaminants (CACs). Activities occurring within the CESI sub-activity are managed by the Highly Energy-Efficient Industrial Science and Technology group (HEIST), which is part of the Office of Energy Research and Development (OERD) within Natural Resources Canada (NRCan). The evaluation encompasses the five-year period from fiscal year 2006-07 to 2010-11 and examines CESI's objectives and activities, which are to support research and development (R&D) to develop cross-cutting technologies applicable to several industrial sectors.

CESI Context and Profile

Canadian industrial activity accounts for 33 percent of the nation’s total energy consumption, 24 percent of GHG emissions and 18 percent of CAC emissions. The great majority of the energy consumption and emissions are concentrated in seven heavy industry sub-sectors (e.g., pulp and paper; mining and smelting; petroleum refining; iron and steel; chemical; cement and lime; and other manufacturing). The CESI sub-activity is meant to work with industry, in the improvement and development of industrial processes and technologies that will reduce industrial energy intensities and lower GHGs and CACs. An additional aim is to ensure that Canadian industry remains productive and competitive.

NRCan is the lead government department responsible for overall program management and administration of funds to research projects addressing CESI’s objectives. Within NRCan, the Office of Energy Research and Development (OERD) plays this overall coordination and management role, including conducting strategic program planning, leading the committee that reviews and selects proposals, allocating resources, and overseeing reporting and performance measurement. The scientists within NRCan’s Innovation and Energy Technology Sector (IETS) implement a large number of the research projects. In addition, the National Research Council Canada (NRC) and Environment Canada (EC) also lead some projects. NRCan’s Office of Energy Efficiency (OEE) has the role of distributing the knowledge gained through its electronic newsletter and annual conference.

All CESI projects are funded through one of the following three components, each of which has a slightly different research focus:

  • the Program of Energy Research and Development (PERD) – ongoing program;
  • the Climate Change Technology Innovation Initiative (CCTII) - ended in 2008; and
  • the ecoENERGY Technology Initiative (ecoETI) – ended in 2011.

Both CCTII and ecoETI target seven industry sub-sectors: iron and steel, mining and smelting, cement and lime, pulp and paper, chemicals production, petroleum refining, and other manufacturing.

Projects funded by PERD are categorized into one of the following research themes:

  • Advanced Process and Technology Integration (addressing energy consuming processes in all seven of the industry sub-sectors);
  • Advanced Industrial Combustion (addressing energy generating technologies in all seven of the industry subsectors); and
  • Transformative Technologies (addressing earlier stage and “riskier” R&D opportunities).

During the five-year evaluation period, the sub-activity received approximately $29 million in NRCan funding to conduct approximately 60 projects. As the majority of CESI projects include external partners (e.g., representatives of the private sector, academia and other research organizations), these external partners are also considered program stakeholders. Partners provided the equivalent of $14 million in cash and in-kind contributions to the projects.

Evaluation Methodology

Three lines of evidence were used in this evaluation: (1) an extensive review of about 250 documents and files collected from NRCan, CESI management and partner organizations, as well as external literature; (2) a total of 32 telephone interviews with internal stakeholders (i.e., individuals who played a significant role in the design and delivery of the CESI sub-activity) and external stakeholders (i.e., project partners and representatives from other S&T organizations); and (3) a detailed review of nine research projects supported by PERD, CCTII or ecoETI funds. This included a review of available documentation, as well as 16 interviews with federal researchers and external partners for each project.

Key Findings

Relevance

The evidence suggests that while there is a clear need to support clean energy R&D for industry, the CESI sub-activity requires adjustments to best fulfill that need. A clean energy research program is warranted in that it can add value to Canadian industry, supporting elements of both international competitiveness and uniquely Canadian energy-efficient solutions. The need to develop innovative breakthroughs and cultivate a close collaboration between industry and government is heightened in the current context of the rapid pace of energy R&D in other Organisation for Economic Development (OECD) countries.The CESI sub-activity, in producing early stage clean energy outputs (such as tools, models, and bench-level technologies), has progressed towards this position.

However, several factors outside the sub-activity’s control indicate that while short-term needs are being met, intermediate and longer term goals linked to technology implementation are not in fact, a current industry priority. For example, clean air regulations for industrial air emissions/energy consumption have not been implemented. As a result of the lack of enforced industrial emissions targets, there is little impetus for industry to make energy efficiency improvements to their technology. In addition, the current economic times have resulted in industry focusing on quarterly results and an increased reluctance to invest in R&D.

In addition, by targeting seven industry sub-sectors and three distinct research themes, the CESI sub-activity is faced with a complex and wide-reaching scope. From the outset, program documentation recognized the difficulty in achieving impacts beyond the project level. This was then underscored by the lack of a systematic means of industry-wide diffusion, as well as portfolio funding and human resources (HR) issues, which decreased the ability of the sub-activity to achieve its full potential.

The CESI sub-activity remains consistent with the mandate and strategic objectives of NRCan, as both focus on the sustainable development of Canada’s natural resources and S&T that bolsters the competitiveness of Canadian industry. The sub-activity is also closely aligned with the federal government’s priorities regarding competitiveness, as outlined in the federal Jobs and Economic Growth Act. Similarly, climate change and air pollution have remained a federal priority over the evaluation period, as seen through the Clean Air Agenda announced in 2006 and renewed in 2011.

NRCan and the federal government play a legitimate and necessary role in clean energy R&D, including that of enabler (through policy), direct R&D performer/funder, and key bridge in the innovation system, linking the primarily fundamental focus of academia with the commercial intent of industry. At the same time, in order to achieve industry-wide impacts, a program like CESI requires a strong dissemination capacity to ensure that knowledge and technology gained on a project level are adequately transferred to the receptor community. The evidence gathered from internal and external stakeholders, and during the project reviews indicates that this role remains unfulfilled, thus limiting the potential for market awareness and cross-sector implementation. It is a role that the government is well-placed to play.

Performance (Effectiveness, Efficiency and Economy)

While the sub-activity has had some success with respect to achieving short-term outcomes, there is limited evidence that it achieved industry-wide intermediate and long-term goals. In addressing short-term outcomes, evidence shows that the sub-activity assisted industry partners in testing new S&T knowledge and contributed to an expansion of the overall knowledge base of energy efficiency and emissions reduction. While the sub-activity was able to implement some clean-energy technologies, resulting in increased capacity and competitiveness for those partners who adopted the changes, this was often on a case-by-case basis.

The sub-activity’s theory suggests that the knowledge gained and the technologies developed through the individual projects conducted will result in energy-efficient and emission-reducing technologies being adopted by industry more generally in the intermediate term. In the long-term, this would then lead to industrial processes becoming more energy efficient and reductions in CAC and GHG emissions being achieved.

The sub-activity does not monitor industry adoption, and the evaluation found limited evidence that knowledge gained from individual projects was being transferred to industry more generally. In short, the evaluation was unable to confirm a formalized transfer of project-level results to the program level or industry adoption of these project-level results. The lack of clear evidence of diffusion of results and take-up by industry puts in question the eventual achievement of emissions reductions and economic gains.

S&T is a long-term process and the exploratory nature of CESI projects may contribute to the lack of evidence towards longer-term outcomes. Program management also believes that some diffusion of CESI results may happen through the efforts of NRCan’s Office of Energy Efficiency and the Innovation and Energy Technology Sector.

Despite these issues, two positive unanticipated outcomes were observed over the evaluation period. Several CESI projects extended the reach and visibility of Canada’s energy research internationally through contributions to an International Energy Agency Implementing Agreement. In addition, NRCan scientists were found to have consistently performed above industry expectations, raising the profile of NRCan among industry partners.

Several factors impeded the sub-activity’s ability to operate in the most economic and efficient fashion. First, problems with the project selection process were observed (e.g., lack of partnering requirement; a closed proposal pool; an imbalanced project selection committee; and an informal project selection guideline) so that the best opportunities for partner leveraging and meeting the broader needs of industry were not addressed. Second, program experience showed that the best projects had taken two or more years in preparation yet several of the projects conducted during the evaluation period were not financed for more than two years. The late release of funds, in some years, also meant that several projects were delayed in acquiring resources or equipment, so that the ability to achieve a significant impact within a short timeframe was further limited. Third, the evaluation found that while the roles of both internal and external stakeholders have been generally well-defined, with respect to CESI, there is an opportunity to improve communication throughout all levels of the NRCan hierarchy, as well as among external stakeholders such as NRC, EC and other project partners. Finally, the evaluation found a fragmented and disjointed approach to performance measurement. Performance targets/indicators were inconsistently presented in program documentation and several issues with performance data collection (e.g., changing reporting formats; inconsistent annual reports; and incomplete financial data) were also observed. This consequently limits the use of the data for informed decision-making and decreases the sub-activity’s ability to determine progress and overall impact.

Conclusions and Recommendations

A federally-funded, clean energy R&D initiative is an important part of the government’s ability to advance promising technologies and strengthen Canada’s technological and economic competitiveness. In this respect, the evaluation confirmed the relevance of the CESI sub-activity as it produced several positive results, and noteworthy project-level outcomes were observed. These include an expanded knowledge base on energy efficiency and emissions reduction indications of improved efficiency among partners who have actually adopted CESI recommendations and are benefitting from increased energy savings. The relevance of this sub-activity is also supported by the national necessity to remain competitive in the energy field through continued public-private S&T partnerships. The evaluation found that the sub-activity is aligned with the current government strategy on curtailing emissions.

The evaluation concludes that despite project-level successes, at the program level the sub-activity as currently designed has a low likelihood of producing impacts related to the intended outcome of industry adoption of new or improved CESI-developed technologies because it lacks a clear diffusion, outreach and knowledge transfer strategy. Consequently, CESI is unable to foster its central aims of industry-wide emissions/energy reductions, competitiveness, or improved industrial capacity through broad awareness.

The evaluation makes four recommendations. Given the challenges the sub-activity has experienced in achieving its energy efficiency outcomes, the first recommendation follows evaluation findings that suggest that a fundamental review of the program should be undertaken. The remaining three recommendations are meant to be taken under advisement in the review of the CESI program.

1. The CESI sub-activity attempts to address three different research themes and seven large industry sub-sectors that account for about 80 percent of Canadian industrial emissions, all within a limited means. Reduced staff and budget over time, combined with two finite funding components and industry reluctance to invest in R&D, counteract the goal of long-term efficiency efforts across a broad industry audience. These factors, as well as the lack of a transfer strategy lead the evaluation to conclude than an overall review of the program is needed.

Recommendation 1: NRCan should review the future direction, design, and scope of CESI, and reassess whether it remains a priority and a viable program, given the need for the department to ensure the most effective use of its limited resources.

2. The sub-activity, as currently designed, has a low likelihood of producing impacts related to the intended outcome of industry adoption of new or improved CESI-developed technologies because it lacks a clear diffusion, outreach and knowledge transfer strategy. Although uptake by individual partners is a starting point, as there is no mechanism for technologies to be transferred to industry as a whole. Consequently, there is little likelihood for adoption at an industry-wide level. Moreover, even if potential emissions reductions and economic gains are being achieved, definitive attribution to the CESI sub-activity is not possible as there is no monitoring of industry adoption. Thus, CESI is unable to fully foster its central aims of industry-wide emissions/energy reductions, competitiveness, or improved industrial capacity through broad awareness.

Recommendation 2: An NRCan CESI program should include a sound strategy to disseminate the knowledge and transfer the technology gained on a project level more broadly across industry sub-sectors, thus facilitating industry-wide uptake and energy efficiency. Such a strategy needs to be developed and implemented collaboratively among NRCan’s Office of Energy Research and Development, Office of Energy Efficiency (OEE) and Innovation and Energy Technology Sector (IETS).

3. The sub-activity supports one-to-one relationships instead of broad network-like partnerships. The evaluation found that leveraging of partner funding was low, few projects included members from other government departments and there was little increase in new collaborations among industrial/institutional partners over the evaluation period. In fact, as the sub-activity did not systematically include a mandatory partnering requirement, the extent to which NRCan researchers could truly engage their external counterparts was limited.

Recommendation 3: An NRCan CESI program should be designed to enhance researcher/partner interactions. As such, NRCan should: a) increase the diversity and number of external partners; b) develop specific criteria that will define the circumstances under which partners will not be required initially, as well as a process for determining when the engagement of partners will be required for these projects; and c) encourage the full contribution of other participating government departments.

4. The evaluation also found that the sub-activity is not utilizing the most efficient or economic means available. An inadequate project selection process, incapacity to support some projects through to fruitful completion, a lack of full-time management resources, inadequate communication both within and outside of NRCan, and inconsistent measurement/monitoring of program impact, all restrict the sub-activity from making sound evidence-based decisions. Thus there is an opportunity to strengthen program implementation and communication, in order to improve decision-making and increase the overall efficiency and effectiveness of the CESI.

Recommendation 4: An NRCan CESI program should include strong project selection processes and performance monitoring.

Recommendations and Management Response and Action Plan Table

Recommendations Management Responses and Action Plans Responsible (Target Date)
1. NRCan should review the future direction, design, and scope of CESI, and reassess whether it remains a priority and a viable program, given the need for the department to ensure the most effective use of its limited resources. ACCEPTED – Strategic Review impacted this portfolio and its ability to deliver on its original goals. NRCan is currently undertaking the development of an Energy Innovation Framework to better define goals and scopes of its programs. As part of this framework approach, the Industry portfolio has been integrated into a broader energy efficiency portfolio which includes building, communities, and transport. This broader grouping of energy efficiency will allow for better strategic investments across this important area of research. OERD will undertake a consultative strategic review of CESI’s activities and goals in order to identify a needs-driven path forward as part of the renewal process for the next Energy Efficiency Portfolio of the PERD cycle (current CESI activities end March 31, 2013). ADM Energy Sector

March 31, 2013
2. An NRCan CESI program should include a sound strategy to disseminate the knowledge and transfer the technology gained on a project level more broadly across industry sub-sectors, thus facilitating industry-wide uptake and energy efficiency. Such a strategy needs to be developed and implemented collaboratively among NRCan’s Office of Energy Research and Development, OEE and Innovation and Energy Technology Sector. ACCEPTED – As part of the portfolio restructuring process, OERD will enhance its collaboration with OEE, IETS and other science-based departments and agencies (SBDAs), in the context of its broad strategic review, to establish a dissemination strategy for the portfolio. It is anticipated that this will continue to include, for example, contributions to OEE’s CIPEC (Canadian Industry Program for Energy Conservation) electronic newsletter as well as the OEE’s bi-annual conference. This process has been carried out informally in the past and will be instituted as a formal part of the new process.
In addition, OERD, in collaboration with OEE and IETS, will examine, and where feasible and appropriate, implement other dissemination mechanisms, such as presentations at trade shows and industry association meetings.
ADM Energy Sector

March 31, 2013
3. An NRCan CESI program should be designed to enhance researcher/partner interactions. As such, NRCan should: a) increase the diversity and number of external partners; b) develop specific criteria that will define the circumstances under which partners will not be required initially, as well as a process for determining when the engagement of partners will be required for these projects; and c) encourage the full contribution of other participating government departments. a) ACCEPTED – OERD will endeavour to increase the diversity and number of external partners through mechanisms such as encouraging multi-partner projects and partnering through industry associations and existing and new consortia.

b) ACCEPTED – OERD will continue its strong focus on collaboration and partnering, and will continue to encourage research projects that include at least one external partner, both individually and through consortia. OERD will develop criteria for determining which projects will not require partners – usually because of their higher-risk and longer-term nature. OERD will also ensure that project plans identify at what stage partners will be required.

c) ACCEPTED – NRCan is leading the development of an Energy Innovation Framework through Deputy Minister-level consultations with other SBDAs. As part of this, opportunities will be taken both to focus and strengthen collaboration with, and leveraging from, key SBDAs that have the capacity to add value to, and benefit from, NRCan's investments in Energy Efficiency.
ADM Energy Sector March 31, 2013
4. An NRCan CESI program should include strong project selection processes and performance monitoring. ACCEPTED – OERD agrees and notes that it has been implementing (from fiscal year 2009-10) standard review, assessment, selection and reporting procedures for all of its programs; these have resulted in improved consistency and reporting (for all portfolios) over the last three years. OERD has completed this standardization of processes for project / portfolio evaluation (and reporting), including providing standard templates and assessment criteria, and providing project proponents with policy and procedures guides. ADM Energy Sector
March 31, 2013

1.0 PROFILE OF THE CESI SUB-ACTIVITY

1.1 Introduction

This report presents the findings of an evaluation conducted on the Clean Energy S&T: Clean Energy Systems for Industry Portfolio (hereafter called the CESI sub-activity). Activities occurring within the CESI sub-activity are managed by the Highly Energy-Efficient Industrial Science and Technology group (HEIST), which is part of the Office of Energy Research and Development (OERD) within NRCan. The sub-activity received approximately $29 million in NRCan funding over the five-year period from fiscal year 2006-07 to 2010-11. The evaluation examines CESI's objectives and activities, which are to support research and development (R&D) to develop cross-cutting technologies applicable to several industrial sectors.Footnote1

1.2 Context

Approximately 25 percent of the Canadian gross domestic product (GDP) can be attributed to activities performed in the industrial sector. In this context, it is not surprising to find that industry (excluding upstream oil and gas) accounts for 33 percent of Canada’s total energy consumption, 24 percent of greenhouse gas (GHG) emissions and 18 percent of criteria air contaminants (CAC) emissions.Footnote3 The great majority of the energy consumption and emissions are concentrated in a number of heavy industries (e.g., pulp and paper; mining and smelting; petroleum refining; iron and steel; chemical; cement and lime; and other manufacturing). As such, R&D in cleaner industrial processes has great potential to contribute to the overall energy efficiency of the sector.Footnote4

1.3 CESI Mandate and Structure

Since 2008, NRCan has operated under three main strategic outcomes. During the evaluation period, the second of these outcomes — Environmental Responsibility — states that Canada is a world leader on environmental responsibility in the development and use of natural resources.Footnote5 Under NRCan’s Program Activity Architecture (PAA), this strategic outcome includes the Clean Energy Science and Technology (S&T) sub-activity (Table 1), of which CESI is a part. It is expected that CESI will work in concert with industry in the improvement and development of industrial processes and technologies that will reduce industrial energy intensities and lower GHGs and CACs. An additional aim is to ensure that Canadian industry remains productive and competitive.

Table 1 Position of Clean Energy Systems for Industry (CESI) within NRCan’s PAA
PAA Level Order Name Expected Results
Strategic Outcome 2 Environmental Responsibility Canada is a world leader on environmental responsibility in the development and use of natural resources.
Program Activity (PA) 2.1 Clean Energy Development and delivery of energy S&T, policies, programs, legislation and regulations to mitigate GHG emissions and to reduce other environmental impacts associated with energy production and use.

Expected results include increased energy efficiency, increased production of low emission energy, and reduced environmental impacts associated with energy production and use.
Sub-Activity (SA) 2.1.3 Clean Energy S&T The advancement of clean energy knowledge and technologies that addresses the needs of partners and stakeholders.
SA Portfolio 2.1.3 Clean Energy Systems for Industry To work with industry in the improvement and development of industrial processes and technologies that will reduce industrial energy intensities and lower industrial air emissions (CAC and GHG emissions) while ensuring that Canadian industry remains productive and competitive.

Source: Natural Resources Canada 2010-11 Program Activity Architecture.

Under the CESI sub-activity, HEIST-managed projects received funding from three different sources:

  • the Program of Energy Research and Development (PERD);Footnote6
  • the Climate Change Technology Innovation Initiative (CCTII); and
  • the ecoENERGY Technology Initiative (ecoETI).

All projects are funded through one of these three components, each of which has a slightly different focus. Projects funded by CCTII are aimed primarily at GHG emissions reductions while ecoETI encompasses both GHG and CACs. Both CCTII and ecoETI target seven industry sub-sectors: iron and steel, mining and smelting, cement and lime, pulp and paper, chemicals production, petroleum refining and other manufacturing.

Projects funded by PERD are categorized into one of the following research themes:Footnote7

  • Advanced Process and Technology Integration (addressing energy consuming processes in all seven of the industry sub-sectors);
  • Advanced Industrial Combustion (addressing energy generating technologies in all seven of the industry subsectors); and
  • Transformative Technologies (addressing earlier stage and “riskier” R&D opportunities).

For the period 2003-08, projects were funded by either PERD or CCTII. PERD funding continued from 2008 and is currently ongoing, while CCTII ended in 2008 and was replaced with ecoETI. All ecoETI projects ended in March 2011.

1.4 Governance and Administration

While NRCan is the lead department for the CESI sub-activity, several organizations within NRCan, as well as other stakeholders play roles at various stages of the program. The roles of each are shown in Table 2 below.

Table 2 Roles of NRCan and Partner Organizations
Organization Manage Program Conduct S&T projects Transfer Knowledge Adopt EE technologies
NRCan OERD LEAD
  • Strategic planning
  • Project review and selection
  • Allocate funding
  • Reporting and performance measurement
LEAD
  • Establish and implement policies for knowledge transfer
NRCan IETS SUPPORT
  • Project review and selection
LEAD
  • S&T for the majority of projects
  • Publish technical papers for journals
  • Make technical presentations at conferences, workshops and training courses
  • Make recommendations for codes and standards
NRCan OEE LEAD
  • Distribute electronic newsletter (Heads-up CIPEC)
  • Annual conference (researchers/ industry)
  • Funds IETS and others to do training courses, workshops and Best Practice Guides
EC, NRC and NRCan-MTL SUPPORT
  • Support project review and selection
LEAD
  • Some S&T projects
External project partners
(industry, academia, research organizations)
SUPPORT
  • Cash or in-kind support
  • Models and methods tested in industry plants
Industry (more broadly)
  • The intent is for industry to become aware of new knowledge and technologies and understand how they could apply to their operations
  • Adopt new EE technologies and processes widely across industry

All CESI activities are managed by the HEIST group within NRCan’s Office of Energy Research and Development (OERD).Footnote8 HEIST was initiated in 2003 to provide continuity for activities being conducted under a number of separate funding sources.Footnote9 The HEIST group encompasses three classes of R&D that warrant federal support:

  • R&D to accelerate the adoption of advanced energy/emissions performance technologies that are economically attractive (one to two year payback) but are unfamiliar to industry;
  • R&D to advance the adoption of advanced energy/emissions performance technologies that are affordable (two to ten year payback) but not sufficiently attractive to compete for capital investment; this R&D will inform government policy and industry with a view to developing the economic incentives to promote adoption; and
  • early stage R&D that will lead to transformative advances in technologies and processes for the industrial sector.Footnote10

CESI is overseen by one or more program leaders appointed from NRCan’s OERD. The leader(s) coordinate and manage the Program at Objective Level (POL) Committee/Expert Group (PCEG) Footnote11, accountable for producing strategic program planning, recommending the selection of CESI projects, allocating resources, and reporting annually on results.Footnote12 This Group is comprised of senior scientific, technical, and policy experts from NRCan as well as from the National Research Council (NRC) and Environment Canada (EC). The number of members has varied (usually between eight and twelve) based on the need for expertise and inter-departmental representation. The PCEG reports to the Director for PERD Program Operations and meets at least twice per year.

The process by which CESI projects are selected involves six steps.

  1. Based on strategic priorities and previous funding rounds, research theme areas are recommended by the CESI program leader(s) and approved by OERD.
  2. Program leader(s) then draft a request for proposals which is distributed by the PCEG to federal science-based departments and agencies.
  3. Researchers submit project proposals, sometimes after consultation with industry and academic partners.
  4. Proposals are ranked by the PCEG using the ProGrid approach Footnote13 (although, in some cases, the committee developed its own criteria for ranking). A pairwise comparison between the top candidates is then completed and the Committee makes its final decision. Recommendations go to the Director General, OERD and/or to the Assistant Deputy Ministers of the Energy Sector/Innovation and Energy Technology Sector (depending on the nature of the project and amount of funding recommended) for approval. Final decisions regarding projects and funding amounts are made by the OERD.
  5. The OERD then notifies the successful candidates and distributes funds to project managers.
  6. Project managers within each organization conducting research are responsible for managing the research activities of the individual projects and reporting back to the PCEG Committee through project status reports and annual workshops. The PCEG uses the status reports to prepare the overall CESI annual report for the OERD.

CESI research-based activities are mostly carried out in federal laboratories at NRCan (i.e. CanmetENERGY at Ottawa and Varennes; and the Minerals and Metals Sector Materials Technology Laboratory – recently relocated to Hamilton). Some projects were carried out by the NRC at the Institute for Chemical Process and Environmental Technology (ICPET) and three projects were run from EC laboratories, within the Canadian Hydraulics Centre.

In addition, the majority of CESI projects include external partners (most often representatives of the private sector, and sometimes academia and other research organizations such as non-profits or research consortia), that provide cash or in-kind contributions to individual projects.

NRCan’s Office of Energy Efficiency (OEE) has the role of disseminating knowledge gained through the CESI sub-activity. The OEE’s electronic newsletter (Heads-up Canadian Industry Program for Energy Conservation – CIPEC) and its annual conference where researchers share their recent findings with the industry were the mechanisms chosen for dissemination.

The targeted beneficiaries of the CESI sub-activity are major industrial facilities across seven industry sub-sectors: iron and steel, mining and smelting, cement and lime, pulp and paper, chemicals production, petroleum refining, and other manufacturing. To achieve the reduction in emissions envisioned by the program, industry must be aware of, and widely adopt the new technologies generated by the sub-activity, thus making their industrial processes more energy efficient.

1.5 Resources and Funding

Funding for CESI amounted to approximately $29 million over five years (2006-07 to 2010-11). Expenditure on salaries plus employee benefit plans was approximately $2 to 2.5 million annually from the three sources for each of the three funding components. The remainder was operation and maintenance (O&M) and other expenditures.

As illustrated in Table 3, PERD funding is ongoing A-Base funding, and its allocation to CESI (approximately $3.4 million per year) is available for renewal every four years. PERD funding was last renewed at the end of 2008-09. The CCTII program ran from 2003 to 2008 and represented an additional $2.2 million per year for CESI during the evaluation period (2006-07 and 2007-08). Following completion of the CCTII, the HEIST group was also tasked with managing the ecoETI’s Low Emissions Industrial Systems (LEIS) portfolio, representing approximately $2.4 million annually for each of the three fiscal years (2008-09 to 2010-11). Like the CCTII, ecoETI funding was not renewed. In addition to the funding provided through these three programs, projects often benefitted from complementary sources of funding (e.g., support from industry or academia, or in-kind contributions).

Table 3 Budget and number of CESI-funded research projects over the evaluation period (2006-07 to 2010-11)
PERD CCTII ecoETI
Period covered by the evaluation 2006-07 to 2010-11 2006-07 to 2007-08 2008-09 to 2010-11
Funding frequency Ongoing (allocated every 4 years) 5 years of funding (complete) 3 years of funding (complete)
Funding type A-base C-base C-base
NRCan CESI funding over evaluation period $17.0 million $4.5million $3.7million
% of CESI funding over evaluation period 67% 18% 15%
Total funding and in-kind by NRCan (CESI and non-CESI) and all partners (industry, university, other levels of government and other) $26.1 million $10.9 million $6.2 million
Number of funded research projects covered by the evaluation 27 projects
(2006-07 to 2010-11)
18 projects
(2006-07 to 2007-08)
13 projects
(2008-09 to 2010-11)
Approximate # FTEs 21 Data not available 25

Source: Compiled from CESI annual reports 2006-2011 and from project reports where gaps existed.

1.6 Logic Model

The CESI logic model (Figure 1) has been developed in consultation with members of CESI mana­gement. Note that this logic model represents an amalgamation of the individual objectives and related activities conducted under PERD, CCTII and ecoETI, based on the different logic models, strategic plans and Results-based Management and Accountability Frameworks (RMAF) provided for the evaluation. One of the limits of this model is that timelines and goals within these documents are not always in accord, even for the same funding component. There is also no clear indication as to which of the outcomes in the present logic model should be specifically fulfilled by PERD, CCTII and/or ecoETI.

The logic model uses a top-down diagrammatic structure to represent the sub-activity’s composition and process flow. The activities presented result in the outputs, over which the sub-activity has direct control. In turn, these are intended to lead to immediate, intermediate and long-term outcomes. The sub-activity does not have direct control over outcomes but is responsible for managing activities to influence their achievement to the extent possible. Note that a feedback loop has been included to show that, for CESI, it may take several iterations of collaborative projects that result in increased knowledge before the research is mature enough to lead to intermediate outcomes, such as the adoption of energy-efficient and emission-reducing technologies by industry.

The timeframe for the present logic model includes expected short-term outcomes in one to three years, four to seven years for intermediate outcomes and eight or more years for long-term outcomes. Footnote14 Within this timeframe, it is expected that PERD projects (funded since 2003) should have reached intermediate outcomes with some progress towards achieving longer term goals. Alternatively, note that both CCTII and ecoETI were sunset programs on a finite funding schedule (see Table 3). As CCTII projects received a maximum of five to six years of funding, it is expected that it may have achieved short-term outcomes and made progress towards intermediate outcomes. As ecoETI is currently in its fourth year of funding, it is unlikely that it has passed through the threshold of short to intermediate outcomes.

Figure 1 Logic model of the CESI Sub-Activity

Source: CESI logic model developed in consultation with members of the CESI management and SED.

2.0 EVALUATION SCOPE, METHODOLOGY AND LIMITATIONS

2.1 Objectives and Scope

The evaluation objective was to assess the relevance and performance of the CESI sub-activity, according to the Treasury Board Secretariat’s Policy on Evaluation, and to provide recommendations as necessary. The evaluation covered the activities that have been supported under the CESI portfolio for the fiscal years 2006-07 to 2010-11.Footnote15 In particular, the evaluation addressed the five "core issues" defined by Treasury Board Canada in the Directive on the Evaluation Function (April 2009). These are:

Relevance of the CESI sub-activity:

  • Issue #1: Continued need for program;
  • Issue #2: Alignment with government priorities; and
  • Issue #3: Alignment with federal roles and responsibilities.

Performance (effectiveness, efficiency and economy) of the CESI sub-activity:

  • Issue #4: Achievement of expected outcomes; and
  • Issue #5: Demonstration of efficiency and economy.

2.2 Methodology

Three lines of evidence were used for this evaluation:

  1. Document, file, and data review: A systematic review was performed of about 250 documents, files and datasets that were collected from NRCan and partner organizations.
  2. Stakeholder interviews: A total of 32 in-depth telephone interviews were conducted with internal stakeholders (i.e., individuals who played a significant role in the design and delivery of the CESI sub-activity) and external stakeholders (i.e., project partners and representatives from other S&T organizations).
  3. Project reviews: Nine research projects supported by PERD, CCTII or ecoETI funds were reviewed in detail. This included a review of available project-level documentation, as well as interviews with both federal researchers and external partners per project. The distribution of projects by selection criteria is presented in Table 4. Further description of the project selection criteria and distribution of interviewed partners is presented in Appendix A.

The distribution of internal and external interviews across lines of evidence is shown in Table 5, while the distribution of interviews across stakeholder groups is presented in Table 6.

Table 4 Distribution of projects by selection criteria
# Funding Project Title Theme/
Sector
Department/
Agency/Lab
Partners
A1 PERD Thermodynamically Guided Modeling and Integration of Separation processes Chemicals NRCan
CANMET Energy
CETC Varennes
Industry
&
Academic
A3 PERD Combined Energy and Water Optimization Pulp and paper NRCan
CANMET Energy
CETC Varennes
Industry
&
Academic
B3 PERD Optimization Tools and Models for Industrial Combustion Processes Mining and smelting NRCan
CANMET Energy
CETC Ottawa
Industry
&
Academic
B4 PERD Efficient Industrial Combustion Furnaces and Processes Mining and smelting NRCan
CANMET Energy
CETC Ottawa
Industry
&
Academic
D4 PERD Optimization of Granular Multi-flow Processes Mining and smelting NRC
Canadian Hydraulics Centre
Industry
&
Academic
IS4 CCTII Reduction of GHG Emissions in the Steel Industry Iron and steel NRCan
CANMET Energy
CETC Ottawa
No partners
MS2 CCTII Advanced Blasting from Comminution Process Optimization Mining and smelting NRCan
CANMET Energy
MMS-MMSL
Industry
E10.
007
ecoETI Sulphur Removal from Fuel Oils used in Heavy Industry Petroleum refining NRCan
CANMET Energy
CETC Ottawa
Industry
E10.
009
ecoETI Development of Innovative Fuel Utilization Approaches to Reducing Energy/Emissions Intensity in Petroleum Refining Industry Petroleum refining NRCan
CANMET Energy
CETC Ottawa
Industry
&
Academic

Source: CESI annual reports 2006-2011 and CESI project status reports.

Table 5 Distribution of interviews across lines of evidence
Instrument Internal External Total
Stakeholder interviews 13 19 32
Project reviews 8 8 (+9*) 16
TOTAL 21 27 48

Note: * Nine interviewees were also conducted in the analysis of project review data.

Table 6 Distribution of interviews across stakeholder groups
Stakeholder group Interviews conducted
Internal 13
NRCan, including OERD and CESI senior management 7
Research and program service managers 3
Departmental laboratory directors/directors general 3
External 19
Industrial partners 11
Academic and other research institutions 6
Non-governmental organizations and other S&T groups 2
TOTAL 32

2.3 Challenges, Limitations and Mitigation Strategies

The following challenges were encountered and mitigated to the extent possible.

  1. Lack of consolidated financial information: A detailed analysis of projects’ use of resources was not possible, as there was no central financial database that includes both amounts of contributions made by NRCan and contributions of partners for each project. Data was compiled from sources such as program strategic planning documents, CESI annual reports, project status reports, partner-provided information and financial summaries provided by NRCan’s Strategic Evaluation Division (SED).
  2. Inconsistent presentation of information over time: Information contained in the program annual reports and project status reports was inconsistently presented. This limited the ability to make systematic comparisons or to identify trends over time (for example, increasing knowledge production or collaborations). To mitigate this, data from the documents was cross-checked as much as possible with other lines of evidence. The Program notes that inconsistency of annual reports was the result of attempts to improve the reporting system annually, resulting in varying information from year to year, including the addition of project-level data that were not initially reported.
  3. Lack of stakeholder database: The evaluation assessment initially proposed an online survey of partners for this evaluation. During the interview stage, it became apparent that contact information was not available for a sufficient number of external partners; thus, the survey could not be completed as planned. Consequently, additional external stakeholder interviews were conducted, and the interview guide was adapted to reflect the initial survey questions that focused on partner needs.
  4. Low participation in interviews and project reviews of representatives from other departments: Securing the participation of representatives of federal departments that participated in the program over the evaluation period presented an additional challenge. Only one representative from NRC, and none from EC, chose to participate in the evaluation. This resulted in an over-representation of NRCan staff in the interview sample. Mitigation involved sampling interviewees from various branches and levels of NRCan; they were asked to provide their perspective on the inter-departmental nature of the program.
  5. Project review selection: The absence of a centralized partner database had an adverse effect on project selection and on the overall robustness of the collected evidence for this method. A key challenge for the project reviews was the selection of the nine projects, which had to: 1) meet the selection criteria (see Appendix A); and 2) have involved both federal researchers and partners who would be available to participate in interviews. The final selection was vetted by CESI management. As the entire pool of external candidates was limited to approximately 40, in order to avoid duplication with those interviewed in the stakeholder interviews, 12 external candidates were considered for the project reviews.
    Securing interviews proved challenging, as many of the contacted partners cited limited involvement in projects, difficulty in recalling specifics and/or the departure of the principal employee involved in the project. This resulted in an over-representation of NRCan laboratories. It was only possible to include one project from NRC and no projects from EC were chosen. It is also noteworthy that the mining and smelting sector was over-represented, possibly as projects in this area were generally large with several partners willing to participate. Other sectors, such as cement and lime and other manufacturing, are not represented in the choice of projects.
    Thus, it is recognized that the findings of the project reviews may not fully reflect the overall performance of CESI projects. Consequently, every effort was made to contextualize information and cross-check the available evidence with interviews and documents whenever possible.
  6. The long-term nature of R&D programs makes them difficult to measure: Often, visible results of science and technology research require a long timeframe to develop. This may be especially true in a field like clean energy R&D which is still in a relatively early stage. The evaluation period is five years, and the expected timeframe in which intermediate outcomes were to be achieved is four to seven years. Long-term outcomes were expected to take more than eight years and in some instances as long as 20 years. This presented challenges for the assessment of the achievement of intermediate and long-term outcomes. Due to the data and stakeholder limitations discussed above, the evaluation was not able to review projects conducted before the evaluation period. Acknowledgement must be made, therefore, that the evaluation can only comment on progress towards long-term objectives in an incomplete manner.

3.0 EVALUATION FINDINGS

3.1Relevance of the CESI Sub-activity

3.1.1 Relevance Issue #1 (Continued need for program)

Evaluation Issues Lines of evidence Assessment
Is there an ongoing need for the program?
  • Document and file review
  • Internal and external interviews
  • Project reviews
Ongoing need for an industry-oriented clean energy research initiative is warranted; however, the CESI sub-activity requires adjustments to best fulfill that need.

Summary: The project-level evidence aptly illustrates that an industry-oriented clean energy research program can add value to Canadian industry, supporting elements of both international competitiveness and uniquely Canadian energy-efficient solutions. The need to develop innovative breakthroughs and cultivate a close collaboration between industry and government is heightened in the current context of the rapid pace of energy R&D in other OECD countries.

The CESI sub-activity, in producing early stage clean energy outputs (such as tools, models, and bench-level technologies), has made some progress towards developing breakthroughs via collaboration. However, the evidence indicates that technology implementation is currently not a major stakeholder priority because industry tends to pull back on R&D under economic duress. In addition to the current unfavourable operating context, the CESI program design is not calibrated to its complex and wide-reaching scope, given its limited financial and human resources.

Thus, the evidence suggests that while there is a clear need to support clean energy R&D for industry, the CESI sub-activity requires adjustments to best fulfill that need.

The following analysis of continued program need is based on a discussion of three points: 1) the ongoing need for a general program that supports industry-oriented clean energy research; 2) the extent to which the CESI sub-activity is addressing particular stakeholder needs; and 3) the continuing need specifically for the CESI sub-activity in its current form.

Ongoing need for an initiative supporting industry-oriented clean energy research: All lines of evidence illustrated that, in general, a clean energy research program can add significant value to Canadian industry, as it facilitates both forward looking energy-efficient solutions and international competitiveness. All interviewed industry partners voiced an urgent need to explore process improvements that support a competitive edge, including energy/emissions efficiency. Concern was expressed that Canada would soon be forced to purchase technologies from other countries, instead of supporting a home-grown industry developing solutions that work in a uniquely Canadian context. This is especially relevant in light of emerging economies like China, increasing production costs in Canada, and the rapid pace of development in Europe.

Despite the recognition that it would be highly advantageous for Canada to position itself as a competitive player in clean energy R&D, both the private and public sectors in Canada are significantly under-investing in energy S&T relative to other OECD countries. A recent bibliometric analysis indicated that Canada nearly doubled its clean energy scientific output in the last five years. However, this growth remains about 12 percentage points lower than the world level, meaning that Canada is inferior to the competition in terms of clean energy R&D research.Footnote16 In addition, according to the Conference Board of Canada’s latest environmental analysis, Canada currently ranks 15th out of 17 countries considered peers by the analysis, for overall GHG emissions per capita, earning a “D” grade. Footnote17 Thus, investing in an initiative to support industry-oriented clean energy R&D satisfies an ongoing need to increase Canada’s global competitiveness while simultaneously increasing the potential to reduce GHG emissions.

Secondary to, but supporting, the competitiveness argument, an ongoing need is demonstrated for a research program, such as CESI, which partners industry with the federal government. Innovative breakthroughs frequently result from close collaboration between industry and government, and this relationship requires time to develop industry buy-in.Footnote18 For example, a recent report on global R&D funding indicates that the private sector lacks the full array of resources to perform clean energy research, while government lacks the means to deploy energy innovation at a large scale. In this context, “collaboration and commercialization are the essential bridges.”Footnote19 This was noted during the CESI project reviews and interviews, as about half of the external partners indicated that their clean energy project would not have proceeded without NRCan’s involvement, and the other half said that the provision of federal expertise/equipment was one of a number of catalyzing factors. In other words, in order to widely deploy competitive clean energy R&D, federal support and a sense of ‘ownership’ from industry are both required, with the recognition that this relationship takes time to cultivate.

Thirdly, federally-supported research projects that have the capacity to extend equipment life while introducing energy-efficient options were seen as highly valuable and warranted continued support. As noted in the program documentation, “major process changes or step changes are rare and typically take 20 years to dominate an industry sector. Due to the substantial investment required in capital equipment, it tends to be used for many years. It is not uncommon for industrial plants to function with equipment that is 50 or more years old. It is generally cheaper to gradually modify, expand and improve existing plants than to build new ones.”Footnote20 This idea was supported in the project reviews. External partners from two industry sub-sectors (pulp and paper, and iron and steel) found that they were increasingly restricted by outdated and inefficient equipment that would cost billions of dollars to replace. In these cases, the need for CESI was clearly shown.

Stakeholder needs and priorities: Despite the general need for a clean energy research program, CESI was limited in the extent that it could meet specific stakeholder needs and align with their priorities in the short-term, because of external factors beyond the sub-activity’s reach. For example, the majority of NRCan interviewees noted that continued program need was directly linked to national emissions legislation. Clean air regulations for industrial air emissions/energy consumption have not been implemented. A Regulatory Framework, including sector-specific compliance plans, was laid out in April 2007 with the intent that draft regulations would be published in the spring of 2008.Footnote21 In March 2008, the Government of Canada published Turning the Corner, an updated regulatory framework providing additional details including an outline of mandatory reductions for industry. These proposed GHG regulations were expected to be finalized in 2009 and come into force on January 1, 2010.Footnote22 To date, regulations have been published for the transport and electricity sectors but not for the industrial sector. The government is currently, “working to realign policies and regulations in order to maintain economic prosperity while protecting the environment and aligning with the United States.”Footnote23

In the absence of federal regulations for industrial emissions over the evaluation period, there is no obligation for industry to prioritize long-term energy efficiency goals over the continuing focus on economic growth. Likewise, in the absence of key federal drivers for the implementation of clean energy solutions (e.g., enforced industrial emissions targets), the relevance of a program that supports development of such solutions may be somewhat limited for industry.

Moreover, reducing emissions and increasing efficiency do not appear to be among the current priorities for industry for a number of reasons. During the case studies, some project partners mentioned that it was common for industry to perform its own research about 20 years ago. However, due to increasingly unfavourable economic influences (e.g., the 2008 recession), R&D is no longer a top industry priority. Compounding this is the fact that energy costs, while rising, are not currently prohibitive. Project partners are less likely to implement large-scale changes without a short-term cost benefit. About half of the industrial partners interviewed noted that their organizations are currently focused on quarterly results, could not currently finance R&D, or required a quick turnaround on project outcomes. In addition, foreign takeovers of Canadian companies, particularly in the iron and steel industry, have precipitated adjustments on both a project level (i.e., original research goals may not align with the new owner’s vision) and on a management level (i.e., time required to establish and develop new working relationships between the federal labs and new partners).

Thus, while the general need for a clean energy R&D program is clear, in the current regulatory and economic context, program goals linked to technology implementation are not an industry priority.

Continued need for CESI in its current form: CESI’s current program structure focuses on individual research projects rather than the need to achieve industry-wide energy efficiency and emissions reduction. The project reviews clearly illustrate that outputs such as tools, models, proof of concept and bench-level technologies are being produced. Some projects are funded to identify bottlenecks in key production steps within a particular industrial sector. Others are developed for a specific partner need based on communication between the federal researcher and the partner. Still, others are developed to respond to industry needs on the basis of consultation with industry experts or preliminary literature reviews that identify likely research opportunities. However, all lines of evidence indicate that longer-term needs, specifically implementation of developed technologies, and sector-wide adoption, that were expected to lead to emissions reduction, are currently not being adequately addressed by the sub-activity.

There was a recognized risk for some time that impacts may only be achieved at the project level. Program documents indicate that because individual R&D projects often focus on one particular process with one industrial partner, “it would be easy to wind up with a portfolio of projects, each with high individual impact, that do not combine to produce an impact at the level of the portfolio.”Footnote24 A concern is that the sub-activity currently lacks systematic means of industry-wide diffusion, limiting the impact of the sub-activity to a few partners involved in specific projects, when the goal is to produce sector-wide emissions reduction and increases in energy efficiency. CESI’s current program structure would need adjustments in order to meet these needs.

An important matter to consider is the expansive scale and complexity of the issues that the CESI sub-activity attempts to address. Given the limited resources of the sub-activity, its mandate may be overly ambitious. As shown in the logic model (Figure 1), PERD projects are aimed at three distinct research areas, while CCTII and ecoETI target the needs of seven large industry sub-sectors that account for about 80 percent of Canadian industrial emissions. More specifically, Canadian emissions come from approximately 45,000 manufacturing plants within these seven sub-sectors, each operating a diverse range of processes associated with the conversion of raw materials and feedstocks into finished products. Of these, NRCan believes that 25 to 30 large companies could become the prime focus of a reduction strategy to achieve the most impact.Footnote25

All lines of evidence also pointed to portfolio funding and HR issues, which may have decreased the ability of the sub-activity to achieve its full potential. A sample of these issues, as described in annual reports, follows:

  1. 2010-11: NRCan performed a strategic review of all of its programs, resulting in the termination of seven of the PERD projects (over the following two years). This — in combination with the fact that the ecoETI Program and all of the R&D projects funded through it came to an end on March 31, 2010 — will greatly reduce the size of the portfolio.
  2. 2009-10: The lead researchers involved in Microwave research for industry retired late in the fiscal year, and EC decided not to continue the research. The portfolio leader retired from NRCan and the alternate portfolio lead moved to another position within OERD. (Note that these two positions were subsequently filled by one lead.)
  3. 2007-08: Uncertainty in the timing of release of funds contributed to challenges in staffing. This required internal adjustment to one CCTII project.
  4. 2006-07: Several projects incurred delays as a result of the late release of CCTII funding, subsequent difficulties in staffing and other unforeseen circumstances. (Note that the program recognized these risks and these projects were given a more thorough review at the Expert Group meeting and were to be monitored against quarterly work plans during the next fiscal year.)

The evidence suggests then that the sub-activity is attempting to address ambitious goals beyond the reach of its small budget, limited human resources, and in the absence of a systematic diffusion and implementation strategy.

3.1.2 Relevance Issue #2 (Alignment with government priorities)

Evaluation Issue Lines of evidence Assessment
Is the program consistent with government priorities and NRCan strategic objectives? Document and file review
Internal and external interviews
The CESI sub-activity is consistent with departmental and government priorities.

Summary: The CESI sub-activity remains consistent with the mandate and strategic objectives of NRCan. It is also closely aligned with the federal government’s priorities regarding competitiveness.

In order to determine overall alignment with government priorities, this section first describes the extent to which the CESI sub-activity is consistent with the mandate of NRCan and then its alignment with federal government priorities.

NRCan’s mandate and objectives: The document review and internal interviews confirm that the CESI sub-activity is closely aligned with NRCan’s mandate and strategic outcomes. Both focus on the sustainable development of Canada’s natural resources and S&T that bolsters the competitiveness of Canadian industry. For example, the CESI sub-activity strives to, “identify, encourage and support the development and application of leading-edge, energy-efficient and environmentally-responsible processes, practices, products, systems and equipment in Canadian industry, to improve its energy efficiency, productivity, competitiveness and profitability, while reducing GHG emissions and other environmental impacts.”Footnote26

Comparatively, NRCan’s mission and mandate encompass the following:

  • the responsible development and use of Canada’s natural resources and the competitiveness of Canada’s natural resource products;
  • policies and program that enhance the contribution of the natural resources sector to the economy and improve the quality of life for all Canadians; and
  • S&T in the field of energy, especially conducting innovative science to generate ideas and transfer technologies.

In addition, the CESI sub-activity falls under NRCan’s second strategic outcomeFootnote28 and is well-aligned in promoting the development of clean energy R&D (see also Table 1).

With respect to federal government priorities, the part of the CESI mandate acknowledging that “Canadian industry remains productive and competitive” is directly reflected in the federal Jobs and Economic Growth Act meant to “secure our country’s economic recovery, encourage growth and create jobs.”Footnote29 Similarly, climate change and air pollution have remained a federal priority over the evaluation period, as seen through the Clean Air Agenda announced in 2006 and renewed in 2011.Footnote30 In summary, CESI currently aligns with federal clean air priorities and objectives to support economic growth and the competitiveness of Canadian industry.

3.1.3 Relevance Issue #3 (Alignment with federal roles and responsibilities)

Evaluation Issue Lines of evidence Assessment
Is there a legitimate, appropriate and necessary federal role in industry-oriented clean energy R&D? Document and file review
Internal and external interviews
Government currently plays a legitimate and necessary role, but is also well placed to play a knowledge /technology transfer role, a role that is currently unfilled.

Summary: NRCan and the federal government play a legitimate and necessary role in clean energy R&D, including that of enabler (through policy), direct R&D performer/funder, and key bridge in the innovation system, linking the primarily fundamental focus of academia with the commercial intent of industry. At the same time, in order to achieve industry-wide impacts, a program like CESI requires a dissemination role to ensure that knowledge and technology gained on a project level are adequately transferred, facilitating market awareness and cross-sector implementation. The evidence indicates that there is a current gap in these types of activities which government is well placed to fill.

Legitimate and necessary role: There is strong documentary evidence to illustrate that the federal government is an essential proponent for the development of industry-oriented clean energy solutions. Broadly, the most important layer for the overall design and strategy of S&T innovation often lies at the level of national governments.Footnote31 There is also a legitimate and necessary role for the Department as expressed in NRCan’s policy of sustainable energy use, as outlined in the Energy Efficiency Act.Footnote32 As described in the 2007 ecoETI Results-based Management and Accountability Framework (RMAF), “public investments are the instrument of choice for mobilizing new solutions across Canada’s innovation chain. Publicly-funded research, demonstration and development (RD&D) to develop appropriate and efficient technology is an important policy tool. It is also a channel for collaborative arrangements and networks; it is essential to achieving the critical results needed to achieve the clean air objectives, providing new long-term impacts that can lever additional investments from provincial/territorial governments and from the private sector.”Footnote33

On the other hand, economic savings gained by implementing energy efficiency measures may not always exceed the cost of implementation especially for large companies. In these cases, public interest alone provides insufficient incentive for the private sector to investigate, identify and develop the technologies that are required. This leaves a clear role for federally-supported R&D to explore opportunities for advanced energy/emissions’ performance and also creates a need for incentives that demonstrate the viability of using the proposed solutions.Footnote34 An early strategic plan for the CCTII makes this case:

“Energy savings may pay for or offset the costs of equipment/processes upgrades, however, the payback period may be too long to attract private sector investment. In a transformed energy economy, the benefits of achieving GHG reductions are given added value by the federal government and are promoted through policy and program support. In the case of opportunities that offer a payback, the federal government can be a more patient investor. Government may only have to provide temporary support to encourage many of the innovations developed […] close cooperation between government policy, and program and industry partners is an important element.”Footnote35

Supporting the documented evidence for a legitimate and necessary federal role, project reviews and stakeholder interviews showed that there are several federal programs with a climate change or renewable energy focus (e.g., Sustainable Development Technology Canada [SDTC], Canadian Industry Program for Energy Conservation [CIPEC], Clean Energy Fund). However, there is little evidence of duplication with CESI, in that none of these programs have the exact same mission or wide-reaching goals. In addition, the majority of internal and external interviewees observed that government labs are able to bridge the gap between the fundamental focus and high turnover of university research, and the concentrated commercial intent of industry. They highlighted that neither industry nor academia can ensure the same kind of continuity that is supported by the long-standing expertise cultivated in the government labs.

Thus, when it comes to clean energy R&D, all lines of evidence indicate that government plays three legitimate and necessary roles: 1) enabler, by driving the national R&D agenda and ensuring that the foundation for innovation is present; 2) direct R&D performer and funder; and 3) catalyst, or key link in the innovation system.Footnote36

Appropriate role: Nonetheless, in order to attain industry-wide impacts, in practice, knowledge gained on a project/activity level must be widely distributed and technology broadly transferred. For example, a recent paper argues that for Canada to overcome a lack of productivity growth, the federal government must, “ensure that technology, once developed, actually gets commercialized.”Footnote37 The literature also suggests that government-supported projects with transferred results are more likely to have, “attributes that provide greater pathways for knowledge spillovers, such as participation in new research joint ventures and connections to a large number of other firms.”Footnote38

The Program indicated that the Office of Energy Efficiency’s (OEE) electronic newsletter (Heads-Up Canadian Industry Program for Energy Conservation – CIPEC) and the OEE’s annual conference where researchers are provided with the opportunity to share their recent findings with the industry were the mechanisms chosen by the sub-activity to achieve technology transfer. A review of recent newsletters and conference programs indicates that CESI research was highlighted in a number of instances. However, the evidence gathered from the interviews and project reviews indicates that stakeholders do not feel that these mechanisms are effective in supporting transfer activities. For example, across all three groups of internal interviewees – i) NRCan and CESI senior management; ii) research and program service managers; iii) departmental laboratory directors/directors general – the comment was made at least once that the CESI sub-activity lacked the ability to transfer the results of promising research to demonstration-scale within industry. Some external interviewees said that the sub-activity was not proactive in terms of partner outreach, facilitating network exchange, or providing guidance towards other opportunities to collaborate. This sentiment was reflected in the project reviews, as about half of the project partners said that more active outreach was required to involve industry partners in the clean energy research process.

Here, it is important to note that in light of the competitive pressure among industry partners and a reluctance to share information among competitors, partners see an appropriate role for government as a neutral knowledge disseminator and network facilitator. However, for many industry partners, their experience with “government” was embodied by the CESI sub-activity which currently lacks a defined dissemination role.

3.2 Performance of the CESI Sub-activity

3.2.1 Performance Issue #4 (Achievement of outcomes)
Evaluation Issue Lines of evidence Assessment
To what extent have intended outcomes been achieved as a result of the program? Document and file review
Internal and external interviews
Project reviews
Short-term outcomes have been moderately achieved. Progress towards intermediate and long-term outcomes is limited.

Summary: The CESI sub-activity has made moderate progress towards achieving its intended short-term outcomes. There is evidence to indicate that the sub-activity assisted industry partners in testing new S&T knowledge and contributed to an expansion of the overall knowledge base of energy efficiency and emissions reduction. Although the sub-activity was able to implement some clean-energy technologies, resulting in increased capacity and competitiveness for those partners that adopted the changes, this was often on a case-by-case basis.

Overall, there is currently limited evidence in support of the sub-activity achieving its industry-wide intermediate and long-term goals. As there is no systematic mechanism for transferring project-level outcomes to the program level or beyond, there is consequently little likelihood that industry-wide impacts are being achieved as envisioned in the logic model. Given this low likelihood, it is also unlikely that the potential emissions reduction and economic gains included in project proposals could have been achieved as a direct result of the sub-activity. As there is no monitoring of industry adoption, only potential reductions/economic gains can be reported rather than sound estimates of actuals, as based on a percentage of CESI technologies/practices implemented. Finally, it should be acknowledged that the nature of R&D, and the “early days” of some of the exploratory CESI projects, also contributed to the lack of evidence towards longer-term outcomes.

To determine the overall extent to which intended outcomes have been achieved, the following analysis is based on a discussion of four short-term outcomes, two intermediate outcomes and two long-term outcomes identified in the CESI logic model.

3.2.1.1Short-term outcomes

Progress towards four individual short-term outcomes ranged from low to very good for an overall moderate achievement. For example, in the first outcome, assisting industry partners test or prove new S&T knowledge, modest progress was observed. Program annual reports do not identify the proportion of projects tested within the government lab or at a partner institution. Thus, measuring assistance in this way was not possible through the document review but was investigated during the interviews. About an equal number of external interviewees cited either a positive or limited impact towards this objective. Those who indicated a positive impact noted that the stage of development and level of assistance varied from new technology tested, to improvement of existing processes, to actual implementation and engineering phases for larger scale development. Those who indicated a limited impact said that it was too soon to make a decision in this regard, or were unable to determine whether the CESI projects helped them to implement a new S&T approach within their organization. Six NRCan interviewees considered progress towards this outcome as moderate, and four as good. Three stressed that although project partnering was strongly encouraged, it was not an explicit requirement. Program documents indicate that 20 percent of all CESI projects included no partners.

Technologies and tools developed in five out of nine reviewed projects were tested in at least one partner organization, with a sixth expected in the near future. These projects spanned several industry sectors including chemicals, pulp and paper, and mining and smelting. In addition to testing technologies at a partner company, assistance also included providing reports and recommendations for ways to implement energy-efficient processes within partner facilities. In eight out of nine CESI projects reviewed, federal researchers recommended ways to:

  • improve operating systems in partner facilities;
  • focus on particular processes for energy optimization, should research continue; and
  • modify old equipment to perform new tasks.

Considering the second short-term objective of leveraging/mobilizing partner funding through government investment, the evaluation found that progress towards this goal was low to moderate. Over the evaluation period, a total of about $44 million was invested in 58 CESI projects.Footnote39 About 60 percent of these investments were made in PERD projects. For every NRCan dollar invested through the three funding programs, a range of 0.4 to 1.7 non-NRCan dollars (i.e., cash and in-kind from industrial and institutional partners) were invested, with an average of about 0.43.Footnote40 Here it is important to note that CCTII was the only CESI-funding component with a compulsory partnering requirement. Nevertheless, partnership was strongly encouraged for all CESI projects and strategic planning documents of all three components (PERD, CCTII, ecoETI) identify the importance of private sector engagement and resources.

Figure 2 depicts the annual cash and in-kind contribution of NRCan and non-NRCan contributions to CESI projects over the evaluation period. Note that non-NRCan contributions generally decreased over the evaluation period, although in the last two years, the in-kind contribution increased slightly. As explained by partners during the project reviews, in-kind contributions most often included the provision of data or expertise, access to the partner site/field conditions, equipment, raw materials, and graduate students.

The leveraging analysis of the nine project reviews echoed the above, showing that for every NRCan dollar invested, a range of 0.0 to 1.3 non-NRCan dollars were provided, with an average of 0.4, excluding one high outlier.Footnote41

  • Overall, this level of leveraging appears low when compared to evaluations of other programs in the Clean Energy S&T sub-activity. For example:
    Clean Electric Power Generation: for each NRCan dollar, 1.13 non-NRCan dollars were contributed;
  • Oil and Gas: for each PERD dollar, 1.5 non-NRCan dollars were contributed; and
  • Transportation Energy: for each NRCan dollar, an estimated 1.12 non-NRCan dollars were contributed. Footnote42

This evidence also supports the suggestion that the relevance of the sub-activity for industry may be somewhat limited. If industry had a clear need for the Program, investment would likely be greater than that observed.

Figure 2 Annual contributions to CESI projects, 2006-2011 (A: NRCan and non-NRCan contributions; B: Nature of investment)

Source: CESI annual reports 2006-2011

The evidence showed that for the third short-term outcome, CESI projects increased the fundamental knowledge base of energy efficiency and emissions reduction to a very good extent but that the output level has declined in the latest years. Documents and project reviews indicated the development of numerous concepts, techniques, sensors and software. In particular, computer models and modeling tools were featured in annual reports and developed in eight out of nine projects reviewed. Projects often sought to understand energy use and identify the key industrial processes that have the most impact on energy consumption and/or emissions. This knowledge is fundamental to understanding industrial energy needs and pinpointing process-specific improvements. Many concrete examples of specific methods and models that increased overall understanding of energy efficiency and emissions reduction were provided in interviews and annual reports (a sample of which is presented in Table 7).

Table 7 Sample of knowledge developed across CESI funding components

Funding component
Research theme/area Example of data/tools/models contributing to an
expanded knowledge base
PERD Advanced combustion technologies Computer application tools were developed including: thermal efficiency evaluation tool; multi-variable controller for combustion processes; generic thermal analyser.
Process integration and Heat Transfer Developed a Thermodynamically Guided Modelling (TGM) design tool for hybrid separation systems used in chemical and petroleum refining.
CCTII Cement and Lime New materials successfully identified as cement substitutes.
Iron and Steel Modeling of the behaviour of an oxyblast furnace with recycling of furnace top gas optimization for CO2 capture, and induration furnace to reduce energy use and GHG emissions through flue gas recirculation.
ecoETI Mining and Smelting Improved understanding of the electric furnace through comprehensive modeling of metal processing.
Chemical Production An innovative approach to catalytic ethane dehydrogenation has led to conversions and yields that were increased to levels higher than that obtained by initial studies using a different catalyst. The approach has shown that membrane assisted catalytic dehydrogenation of ethane can improve process efficiency to the point of commercial viability.

Source: CESI annual reports 2006-2011 and CESI project status reports

Table 8 shows the number of outputs in which program-created knowledge was diffused. These include client reports, scientific papers, presentations, knowledge transfer products, and patents. Across all 58 CESI projects conducted during the evaluation period, documents indicate that for the first three years of the evaluation period (2006-2009), at least one client report was produced per project. Overall, comparing the total number of these outputs as reported for the earliest two years of the sub-activity (355) to the latest two years (167), the average went from about six (6) outputs per project per year down to about three (3). While changes in reporting, the scaling down of CESI following the end of CCTII and eco-ETI, and the reduction in the number of projects as a result of the strategic review may have also played a role, there is an indication that the outreach of the sub-activity has decreased significantly.

Table 8 Sample of outputs contributing to increased knowledge base over evaluation period
Outputs 2006-07 2007-08 2008-09 2009-10 2010-11
Number of projects 28 32 26 26 26*
Client reports 29 44 45 10 19
Refereed/peer reviewed publications 41 55 23 18 25
National/International presentations 73 92 139 26 36
Knowledge transfer products 4 11 6 5 8
Patents issued to NRCan 1 2 2 4 6
Reported outputs 149 206 241 67 100

Source: CESI annual reports 2006-2011.
Notes:* Seven projects were cancelled so that the number of ongoing projects dropped from 26 to 19 at some point during the year.

Finally, considering the fourth short-term outcome, existing collaborations among federal participants, industry representatives and other research institutions were strengthened to a good extent under the CESI sub-activity. However, this result was tempered in that progress towards a new or expanded network of collaborators in the recent years was found to be low. To illustrate, all available data on project partners from 2004 to date was compiled. A total of 190 partnerships were counted with 108 unique organizations. However, as shown in Table 9, the number of new partners decreased steadily over time. That is, while partnerships were still being made, they were frequently made with the same organizations as in previous years and rarely with new organizations.

Table 9 Collaborations with new partners over the evaluation period

Partners (industry + institutional)

PERD (2004-08)

CCTII (2004-08)

ecoETI (2008-11)

PERD (2009-11)

# of partnerships /funding component

64

40

20

18

# of new partners

64

24

12

6

This trend was also seen in the project reviews, as six out of nine projects stemmed from prior collaborations, and four of those expected to maintain a future collaboration. Only two out of nine reviewed projects included a new, first-time collaboration. The majority of partners interviewed in the context of the project reviews and during the external stakeholder interviews indicated that the federal laboratory was the first to initiate a project idea or invitation to collaborate. Some said that the decision to collaborate was mutual after meeting through a third party (such as another partner or industry association), at a workshop or seminar, or through another government funding program such as Canadian Industry Program for Energy Conservation (CIPEC)Footnote43 or NRC’s Industrial Research Assistance Program (IRAP). Less commonly, the partner contacted the federal government with a specific need or research question.

As in the case of the partners’ contribution to the CESI sub-activity, this evidence, while supporting the examination of the performance of the sub-activity, also goes to indirectly support the suggestion that the relevance of the sub-activity for industry may be somewhat limited.

Two items were often commented upon as elements that could detract from continued project participation. First, some partners noted a disconnect between industry’s needs for rapid solutions versus the perceived long development time in government labs. Secondly, as described by some interviewed partners, collaboration with government was perceived as being associated with more paperwork to complete, federal jargon to decipher, and/or a longer or more rigorous approval process to purchase equipment, modify research scope, etc. This issue of a higher administrative burden was also raised in two of the project reviews where partners expressed frustration as they followed federal protocols (for contract amendment and software licensing) that were not always clear or swift. With respect to both of these barriers, it is recognized that these are commonly-held perceptions within industry, and the CESI sub-activity may not be in a position to entirely overcome these kinds of external opinions. As they reflect a general perception of participation in government programs, these factors alone cannot explain why the CESI sub-activity is increasingly facing difficulties in attracting new partners, yet they should be addressed in order to increase industry participation.

Both internal and external interviewees observed that partnerships were often limited in reach. For example, the majority of internal interviewees said that collaboration with industrial companies often occurred on a one-to-one basis instead of in a network that included several manufacturers across an entire industry sector. Similarly, many external partners were seeking benefits from these one-on-one relationships (e.g. links to more partners extending from the original alliance; or a way to access information from other projects) that would normally be associated with a network-type collaboration.

Importantly also, interview evidence suggests that while federal researchers may have access to existing R&D clientele through various NRCan laboratories, this information is not centralized in a database of partner contacts. This, combined with staff turnover in the participating federal departments, curtails the internal capacity to access partners and create new collaborations.

3.2.1.2 Intermediate outcomes

In order to discuss the intermediate outcome, adoption of efficient technologies by industry, a brief contextual explanation is required. The CESI sub-activity is meant to produce industry-wide impacts on the basis of knowledge gained through individual projects. In practice, there is little evidence that indicates that impacts are being achieved beyond the project level. Although uptake by individual partners is a starting point, as there is no mechanism for technologies to be transferred to industry as a whole, therefore there is little likelihood for adoption at an industry-wide level. Finally, even if progress was being made in this respect, definitive attribution to the CESI sub-activity is not possible as there is little measurement of wider uptake.

All lines of evidence indicate that the sub-activity generally supports individual partnerships and that there are clear successes of individual partners adopting new technologies. For example, the project reviews and documents indicate that adopted technologies by individual partners span at least four out of the seven CCTII/ecoETI targeted industry sub-sectors and three of the four PERD research themes (see Table 10 for examples).

Table 10 Sample of technologies adopted at partner organizations
Funding component Research theme/area Technology adopted
PERD Advanced combustion technologies A glass manufacturer installed a burner for more efficient pet coke combustion.
Advanced sensors and controls A software tool for more energy-efficient lumber drying was installed at two plants.
Process integration and Heat Transfer Modeling tools were adopted and applied to the design of a Pressure Swing Adsorption (PSA) system and to the design of industrial C2 and C3 splitters, leading to an energy efficiency pilot by a major chemicals producer.
CCTII and
ecoETI
Pulp and Paper Flue gas sampling system at Norampac Pulp Mill was overhauled and the sample conditioning system was redesigned and rebuilt for additional tests.
Iron and Steel A model generated by CANMET showed that significant savings in energy and GHG emissions are possible through flue gas recirculation for the QCM facility.
Mining and Smelting In a project investigating the rock breakage energy from blasting to comminution utilizing electronic blasting detonator technology, the electronic detonator technology was adopted at Rocky Lake Quarry.
Chemical Production Binary hybrid distillations systems were retrofitted at a partner company using TGM methodology developed during the PERD project (listed below under the process integration category).

Source: CESI annual reports 2006-2011 and CESI project status reports.

While an early strategic plan for the HEIST program recognized the risk in operating without a defined transfer mechanism,Footnote44 the 2007-08 annual report notes that when the strategic plan was first prepared, demonstration activities were supported under two specific programs outside of the CESI (these were the Technology Early Action Measures [TEAM] Program, and the Industry Energy Research and Development [IERD] Program). As both programs had since ended, the report goes on to state that, “the absence of such demonstration programs leaves a gap that is particularly significant in the case of industry and the HEIST program.” Furthermore, “the ecoETI program does not include a demonstration component for industry. For HEIST, this leaves a gap in the delivery of impacts that is a concern for the HEIST program.”Footnote45

To address this gap, program documentation suggested that project proposals should include a well-defined technology transfer plan.Footnote46 Of the almost 60 projects conducted during the evaluation period, only nine project proposals were provided for examination (for the nine conducted project reviews). Of these nine proposals, only four included any description under “dissemination strategy.” Three of these were vague or generalized statements such as, “Research results will be disseminated through industry. Client involvement to plan research...” or, “There will be seminars and presentations on this work to industry, the scientific community and policy people.” Only one of the nine provided a concrete dissemination strategy detailing:

  • the methods to be used to introduce the proposed tool to the project partner;
  • an estimate of the project leader’s time (about 10 percent) to be spent promoting the technology to other (non-project) industrial contacts;
  • a research plan to source additional funding options; and
  • a presentation and promotion strategy at national/international conferences.

It is also important to note that the program structure which supports individual partnerships also creates challenges relating to competition among individual firms within an industry. A few stakeholders interviewed spoke of a reluctance to share information among competitors and difficulties relating to sharing of intellectual property as factors that may have hindered uptake.

Finally, even if industry-wide adoption were to occur, the sub-activity would be unable to determine this as there is little tracking of this type of outcome. For example, none of the annual reports present the number or percentage of projects that have resulted in technology uptake by partners. Neither do the majority of documents present an estimate (or an easy way to estimate) the level of adoption of clean energy technologies or practices, by individual industry partners or more broadly within industry sub-sectors. For example, only the most recent set of project status reports (2009-2011) include a relevant indicator, namely a check-box for “results used by other stakeholders to conduct RD&D.” This is the first time an indication of use by external partners has concretely been identified in the reporting structure. A manual count of the 38 status reports provided for these years indicates that only five (about 13 percent) have a checkmark/description under this indicator.

A second intermediate outcome was greater capacity to develop and implement energy-efficient processes through communication and awareness. The importance of communication and awareness to capacity building was recognized in CESI sub-activity documentation. For example, the 2009-13 Program Planning Document states that, “through an extensive network of connections to private industry, the group ensures engagement and awareness on the part of the industrial adopter community.”Footnote47 However, the document offers little explanation of how an extensive network will be built, and there is little evidence that illustrates efforts to actively foster broader awareness among the industrial community via the CESI sub-activity. In this respect, a few internal interviewees pointed to the Heads-Up Energy Efficiency Newsletter, circulated by the NRCan Office of Energy Efficiency (OEE). With a monthly circulation of about 12,000 subscribers, the newsletter has the broad mandate to inform “about energy efficiency issues and initiatives in commercial, institutional and federal government buildings.” While successful CESI projects are described in the newsletter, there is no evidence to suggest that this is a mechanism specifically chosen by the CESI sub-activity to create awareness. As well, it may not in fact be appropriate to the sub-activity’s needs as the effect of these newsletters has not been monitored.

Many external partners noted a general lack of knowledge about the CESI sub-activity (outside of their own individual involvement), suggesting that awareness and communication need to be improved. Notwithstanding, the project reviews and annual reports do offer case-by-case evidence of improved capacity by those partners who identified new opportunities or capabilities through a CESI sub-activity project. Some internal and external interviewees cited signs of industry communication and uptake, in that a few of the program-developed results are being used by companies that were not initial collaborators. During the investigation into the project reviews, this was found to be the case in two (out of nine) projects. These two projects involved participation from an industry association (COREM in the Mining and Smelting sub-sector, and CCRA in the Iron and Steel sub-sector). This may suggest a role for industry associations in promoting awareness and improving sector-wide capacity building.

3.2.1.3 Long-term outcomes

Before discussing long-term outcomes, it must be acknowledged that the nature of R&D often makes it difficult to assess long-term impacts. Nevertheless, this evaluation sought to examine the CESI sub-activity’s contribution towards industrial processes that are more energy efficient through a reduction of CACs and GHGs and to determine if the sub-activity increased the competitiveness of the Canadian industry. To this end, a prior NRCan Clean Energy S&T evaluation noted, “GHG reductions and economic impacts can only be achieved to the extent that the technologies and knowledge are ultimately commercialized by industry.”Footnote48 Since there is little likelihood that industry-wide adoption is happening as a result of this sub-activity (see short-term and intermediate outcomes), the sub-activity cannot be credited for any large-scale emission reductions that happened.Footnote49

Furthermore, though an estimate of actual reductions/generation of economic activities could be achieved if the adoption of CESI technologies/practices were adequately tracked, considering the current tracking mechanisms, it is difficult to link results to specific projects. Therefore, subsequent attribution of economic and emissions impacts, at an industry-wide level, is not currently possible. For example, the sub-activity uses a tool known as Scope Tables to map GHGs, CACs and fuel consumption to key production steps in the seven industry sub-sectors.Footnote50 In fact, part of the project selection criteria is the ability to address these key production steps. However, the identification of potential reductions is inconsistent across project funding applications, i.e.:

  • CCTII applications include a section in which the applicant must detail expected emissions/energy benefits corresponding to the Scope Tables;
  • ecoETI application forms do not include a specific Scope Table section; instead, relevant information on potential reductions was left to the applicant’s discretion and usually provided in the project description; and
  • for PERD, the project application forms ask the applicant to complete a “potential GHG Reduction Calculation spreadsheet.”

Beyond this initial step of identifying potential reductions, once funds are awarded, annual project status reports do not consistently map performance back to the original targets in the project applications. Notably, the status report templates do not include a quantitative section for emissions reduction, so as each project leader manually completes reports, inconsistencies are apparent.

When rolled up at the program level, the CESI sub-activity annual reports do indicate which Scope Table targets have been addressed, and progress towards CAC or GHG emissions is reported as potential progress, rather than actual progress. For example, the 2010-11 annual report described ecoETI projects thus (emphasis added):

“Collectively, the 12 projects have addressed 25 of the 50 energy and emissions targets identified in the Scope Tables. The projects have generated knowledge, technology and tools that address 21 Mt/a in potential GHG reductions (12 percent of Canada’s industrial GHG emissions). In achieving these climate change impacts, the projects also address potential for 320 PJ/a reduction in energy consumption (12 percent of industrial energy consumption).”

As stated previously, if the sub-activity was able to track adoption and industry use of CESI technologies, estimates of actual reductions based on these potentials could then be made. Thus, currently, the CESI sub-activity can mostly report potential impacts, but it lacks the knowledge diffusion and technology transfer mechanisms to convert a sizeable portion of this potential into actual impacts. It also lacks robust ways of monitoring reductions in emissions.

Similar logic is applied to the ability to measure an increase in the competitiveness of Canadian industry. Again, the economic gains, real or potential, experienced by industry partners as a result of CESI projects have not been measured nor documented by the sub-activity. Furthermore, statistics on intellectual property (IP) gains for industry partners under the CESI sub-activity have been collected only at the most basic level. That is, annual reports tally the “number of patents” but there is no description of how IP is managed, who holds the IP rights (NRCan or partner), or whether that IP is generating an economic benefit for the sub-activity. Therefore, competitiveness for the various industry sectors under CESI can best be determined by looking at cases in which partners adopted energy/emissions performance technologies that are economically attractive.

As shown in Table 11, in total, four of the nine project reviews included some element of increased competitiveness, either for the partner organization or for NRCan. Three of the nine projects reviewed (from three different sub-sectors: chemicals; pulp and paper; mining and smelting) indicated that reduced energy and water consumption is expected to lead to cost savings and increased competitiveness for the industry partners involved. Of these three, only the pulp and paper project estimated the cost, indicating that a 15 percent energy reduction would translate to about $5 million in annual savings.

Table 11 Competitive outputs observed in four out of nine reviewed CESI projects (sector)

Competitive outputs
PERD A1 (chemicals)
Thermodynamically Guided Modelling and Integration of Separation Processes
PERD A3
(pulp and paper) Combined Energy and Water Optimization
PERD B4
(mining and smelting)
Efficient Industrial Combustion Furnaces and Processes
CCTII MS2
(mining and smelting)
Advanced Blasting from Comminution Process Optimization
Energy/water reduction leading to increased cost savings
Software developed by NRCan, commercialized, and available to industry
Patents (IP rights held by NRCan)

Source: Project review interviews and annual project status reports.

Finally it should also be recognized that even when projects are successful, introducing widespread change requires substantial time. Regarding this point, most internal interviewees noted that long-term progress was very difficult to measure because: a) the time frame was too short to make any valuable observations; or b) there is no project follow-up after completion. Thus, the absence of a formal project follow-up strategy is compounded by the “early days” nature of several of the CESI projects and this factor should not be overlooked.

3.2.2 Performance Issue #4 (Unintended outcomes)

Evaluation Issue Lines of evidence Assessment
Have there been any unintended (positive or negative) outcomes? Document and file review
Internal and external interviews
Project reviews
Two positive unintended outcomes included increased Canadian visibility in international energy research and improved recognition for NRCan among industry partners.

Summary: Two positive unanticipated outcomes were observed over the evaluation period. First, through contributions to an International Energy Agency Implementing Agreement, several CESI projects extended the reach and visibility of Canada’s energy research internationally. Second, as NRCan scientists consistently performed above industry expectations, they raised the profile of NRCan among industry partners.

Two positive unanticipated outcomes resulted from the CESI sub-activity. Some program planning documents recognize that the scope of achieving industrial energy efficiency improvements through process innovation, in several intensive sub-sectors, is significant in comparison to the R&D resources that are available to Canada. The documents note, “in reality, some of the energy emissions challenges may be beyond the resources of any single nation” necessitating international collaboration and discouraging isolated national efforts.Footnote51 To this end, the evaluation identified about ten CESI projects that included an element of national and international policy-related research. More specifically, these projects involved a Canadian contribution to an International Energy Agency (IEA) Implementing Agreement. According to IEA literature, as well as the interviewees from the Other S&T Group, these technology agreements are a “flexible and effective framework for IEA members and non-members to carry out deployment or demonstration programs” and to encourage technology-related activities that support energy security, economic growth and environmental protection.Footnote52

Secondly, across all project reviews, and during external interviews, the majority of industrial and other partners indicated that the CESI sub-activity exceeded expectations in providing easy and open access to unique government facilities and highly qualified personnel. This, in turn, enhanced their willingness to collaborate in the future, and bodes well for the visibility and/or reputation of NRCan among external stakeholders. Considering Canada’s need to remain internationally competitive and connected to key industrial players, both unanticipated performance-level outcomes support the suggestion that a federal clean energy R&D program is relevant.

3.2.3 Performance Issue #5 (Demonstration of Efficiency and Economy)

Evaluation Issue Lines of evidence Assessment
Is the program the most economic and efficient means of achieving outputs and progress towards outcomes? Document and file review
Internal and external interviews
Project reviews
Improved processes for project selection, internal/external communication and performance monitoring could enhance program efficiency and cost-effectiveness.

Summary: The evaluation found that partner participation was strongly contingent on the demonstrated ability of the CESI sub-activity to identify and address partner needs such as access to specialized expertise, ability to cost-share, and provision of technical support. Aside from this, several specific factors limited the ability of the sub-activity to operate in the most economic and efficient fashion. First, the evaluation found that there is an opportunity to improve the project selection process so that those projects with the best ability to meet the needs, and that have an impact on industry more broadly are selected. Secondly, the evaluation found that efficiency could be improved by enhancing communication both within NRCan (i.e. clarification of the roles played by the OERD, OEE and IETS) and outside NRCan (i.e., enhancing dialogue with other federal departments and external stakeholders). Lastly, the strategies for performance measurement and monitoring are inconsistent. This limits the use of the data for informed decision-making and also limits the ability of the sub-activity to determine its progress and its overall impact.

The evaluation found that several factors affected the efficiency and economy of the CESI sub-activity. This section begins with a discussion of the positive influences and then turns to areas for improvement.

Interviewed external partners were asked to comment on factors that facilitate participation in CESI projects. As may be expected, participation was strongly contingent on the demonstrated ability of the sub-activity to identify and address specific partner needs. For example, key facilitators (listed from most to least frequently-cited) were:

  • specialized and valuable expertise from federal researchers;
  • ability to cost-share project resources and time;
  • provision of on-site technical support for partner staff or students;
  • compatibility of research results with existing partner processes/machinery;
  • meaningful and ongoing communication amongst partners throughout the project; and
  • provision of usable results within the given timeframe.

Some internal interviewees also cited the following facilitators:

  • the continuous nature of the PERD funding cycle which allowed NRCan scientists some assurance that a mid-cycle project idea could be explored in the upcoming funding cycle;
  • a strong traditional tie between federal labs and industrial partners, useful for initiating project collaborations; and
  • the complementarity of partner/researcher expertise which enables efficient project coordination/communication.

Despite these key enabling factors, issues under four themes were found to have impeded the efficiency and economy of the sub-activity. These themes were: 1) the project selection process; 2) incapacity to bring some of the projects to a fruitful completion due in part to funding issues; 3) internal and external communication; and 4) performance measurement,

The project selection process: One method to assess efficiency and cost-effectiveness is to examine the extent to which the sub-activity is capable of leveraging investment from other sources. CESI’s leveraging capability was low when compared to other programs in NRCan’s Clean Energy S&T portfolio. Evaluations of other Clean Energy S&T sub-activities indicate that leveraging ranged from 1.12 to 1.5 non-NRCan dollars for these programs, as compared to an average of 0.43 for CESI. Importantly, however, there was some indication that projects which managed to diffuse results to end-users who were not the initial collaborators were those that included either industry associations or more than one large industrial partner. A similar observation was made in a 2007 internal document which stated that in order to achieve impact in industry, the sub-activity “needs to work with the receptor community to identify best opportunities for R&D and demonstration,” and to “initiate a dialogue with the industry community to identify opportunities that have an impact and that they will partner in.”Footnote53

Several issues around the project selection process directly contrast with the above-stated intent of working with industry. For example, and although encouraged, there was no compulsory partnering component for PERD or ecoETI projects. Additionally, as indicated by many interviewees, the small CESI budget necessarily resulted in a closed project selection process among only the three participating departments (i.e. NRCan, EC, NRC), thus reducing the overall potential for a competitive proposal pool.

Furthermore, as illustrated in the annual reports, the CESI Portfolio Committee (i.e. project selection committee) was usually composed of about seven NRCan members and two members of other departments over the evaluation period. Sub-activity management indicated that although the invitation has been extended, it has been difficult to maintain representatives from other departments. While unintended, this over-representation of NRCan clearly presents a potential for biases in project selection. A related concern was that the project selection committee only uses an informal guideline. Some interviewees said that terms of reference outlining defined roles and responsibilities, and criteria for membership selection should be formalized to avoid conflicts of interest or over-representation of any one department. All of these factors have contributed to the project selection process having limited the sub-activity’s ability to enhance the diversity of proposals put forward and maximize partners’ investment in selected projects.

Incapacity to bring some of the projects to a fruitful completion due to funding issues: The potential for reduced effectiveness of the sub-activity through an imperfect project selection mechanism was compounded by a lack of ability to sustain the funding of some ongoing projects for a sufficiently long period. For instance, the aforementioned 2007 internal document also noted that CESI’s experience up to that point showed, “that the best projects had taken two and more years in the preparation. This is not surprising given the size of the partners [CESI] wishes to attract and given the inertia and reticence of large capital intensive firms.” In contrast, the program financial documents and annual reports show that some ecoETI projects were not funded for more than two years and seven of the most recent PERD projects are to be discontinued not having reached two full years of funding. Over the evaluation period, the late release of funds (detailed in annual reports 2006-07, 2007-08 and 2008-09) also meant that several projects were delayed in acquiring resources or equipment, so that the ability to achieve a significant impact within a short timeframe was further limited.

Internal Communication: Ensuring that roles and responsibilities are clearly defined, communicated, and understood both within and outside of NRCan is another important aspect enabling efficient and economic operation. To this end, internal roles have been defined across the various components of the CESI sub-activity. For example, the ecoETI RMAF (2007) and the program planning documents (HEIST POL Plan 2005-06 to 2008-09 and HEIST Program Planning document 2009-10 to 2012-13) clearly outline responsibilities for key stakeholders including the Assistant Deputy Minister’s (ADM) Panel, a Director-General Standing Committee, the OERD, the Portfolio Committee, Project Managers and funding recipients.

More generally, the internal roles and responsibilities of the Department are also represented by:

  • the Office of Energy Research and Development (OERD), the Government of Canada's co-ordinator of energy research and development (R&D) activities;Footnote54 OERD coordinates and manages CESI and the funding allocation process;
  • the Innovation and Energy Technology Sector (IETS) which is the principal federal performer of energy R&D housing the scientific expertise on clean energy technologies;Footnote55 with respect to CESI, members of IETS (specifically the CANMET laboratories) are often funding recipients performing the projects; and
  • the Office of Energy Efficiency (OEE), described as a center of excellence for energy efficiency and alternative fuels information, is mandated to strengthen and expand Canada’s commitment to energy efficiency;Footnote56 with respect to CESI, the OEE may play a peripheral role and is only mentioned very briefly in the CESI planning documents.Footnote57

Nonetheless, there may be an opportunity to revisit communication of CESI roles and responsibilities within the OERD, within IETS, and across OERD, IETS, and OEE, with an eye towards improvement. For example, the majority of internal interviewees said that clearer communication of how CESI fits into strategic priorities, set from the level of ADM and up, was required while half of the internal interviewees said that it was not always clear how decisions were made from the OERD level and higher. These interviewees also pointed out that few people outside the sub-activity know what it is about. Many interviewees, and one internal document,Footnote58 mentioned that to enhance outward communication, the CESI sub-activity was at one point seeking to engage in a more synergistic working relationship with the OEE. However, according to interviewees, this key linkage was never formalized possibly because of the potential for increased administration and reporting burden, combined with decreasing staff levels and already stretched resources. Note that no OEE representative was available to be interviewed although several attempts were made.

Considering IETS, eight of the nine reviewed projects included documentation from and interviews with, project leaders/scientists representing this division. The provided project proposals and status reports show great variability in the level and consistency of completion. Recognizing that laboratory managers and project leaders have several significant responsibilities outside of the CESI sub-activity, there is nonetheless an opportunity to work on standardization in reporting. At present, the lack of consistency hinders upward communication because the compilation of incomplete or inconsistent project data makes it difficult to generate dependable annual reports. This, in turn, impedes the ability for the sub-activity to accurately analyze its achievements over time and overall accountability is thus decreased.

With regard to the communication between IETS and OERD, about half of the interviewed project leaders/scientists indicated that over the evaluation period, they had had some direct contact with OERD management through the Program, and praised them for prompt and clear direction. The other half stated insufficient experience with the OERD to comment or expressed a need for more frequent and meaningful feedback to both strengthen the research direction, and to help researchers understand what is required of them from a management perspective.

Here, it should be noted that an opportunity for feedback and communication between IETS and OERD is addressed through an annual CESI workshop. At this meeting, project leaders are invited to present results to their colleagues, members of the CESI Portfolio Committee and any invited partners. However, because of time and budget constraints, this workshop did not occur in 2011.

External Communication: Many internal interviewees noted that they were disappointed to observe that in recent years, generally, the number of meetings, presentations and workshops has significantly decreased and some noted that external partners rarely attend the CESI annual workshop. Some internal interviewees also noted that communication among other government departments, such as NRC and EC, has declined over the evaluation period. While CESI aims to be multi-departmental, only one of the nine project reviews included a collaborator from another department, and documents indicate that this occurs in less than a quarter of projects. Thus, the opportunity to leverage the already established partner networks and S&T knowledge of other departments or to promote CESI to external stakeholders has also decreased.

Performance measurement: Sub-activity managers are responsible for determining appropriate ways to measure their outputs and progress towards their outcomes, collecting information against those measures, monitoring overall progress, and using that information to inform program decisions. As alluded to previously, the evaluation found a fragmented approach to performance measurement over the evaluation period. While attempts were made to develop systematic performance measurement tools, these were either not completed, were focused at outputs, or changed over time.
Chronologically reviewing the documents, a risk-based audit framework for CCTIIFootnote59 indicated that monitoring would be performed according to the, “targets and indicators set out in the Results-based Management and Accountability Framework (RMAF.)” While a draft version of this RMAF was created, internal program correspondence indicates that, in fact, the CCTII RMAF “was never finalized” and, instead, a strategic planning document was used for performance monitoring purposes. This document outlines the context and logic model for the Initiative as well as expected outcomes, but no performance indicators or monitoring strategy are presented.Footnote60

Later, the POL Plan 2005/06 to 2008/09Footnote61 presented performance targets for immediate (e.g., primary industries and suppliers are aware of developments) and intermediate (e.g., by 2020, 50 percent market penetration of technologies adopted) outcomes of the PERD research themes. The 2007 to 2011 HEIST strategic plan went a step further, creating an Impact Potential Matrix (IPM) that compiled all Scope Tables to come up with potential GHG/CAC/energy impacts across the entire portfolio. The document states, “It is not possible to claim potential for reductions as actual reductions. […] Going forward, it is to be expected that some portion of those [potential] reductions will be realized as actual reductions, and this portfolio will be a contributor to those reductions.”

More recently, at the departmental level, a 2010-11 PAA performance measurement framework suggested to merge several sub-sub-activities into the overall Clean Energy S&T sub-activity. The document stated that, “the elimination of these sub-sub activities [including CESI] will result in more efficient management and, combined with a refinement of the performance management framework, will improve our reporting and management.” This document indicated that the main performance indicators are the number of partnerships to research, develop and deploy technology, as well as the availability of scientific knowledge to clients and partners. These are to be collected annually through lab reports, surveys and program reports with a target to maintain the previous year’s number.Footnote62 Thus, while the documents did outline performance measures, they were inconsistently presented, changed frequently and were consequently difficult to follow. In addition, measures were not developed beyond the immediate outcome level.

Similarly, all lines of evidence indicate that performance data collection is inconsistent and that the monitoring plans are inadequate to effectively demonstrate achievement of expected outcomes. A number of individual issues were observed, as outlined below.

  • Project status reports: Project status reports are the means by which the sub-activity collects information at the individual project level. Over the evaluation period, the template format changed, limiting the capacity to compare results over time. The template makes no distinction between the various “stages” of research (i.e., feasibility vs. demonstration studies). Only the most recent set of project status reports (2009-2011) include a relevant indicator relating to adoption, namely a check-box for “results used by other stakeholders to conduct RD&D”. This approach does not provide sufficient information to assess how widely the results of the research were used by industry. In addition, beyond the initial step of identifying potential energy reductions, once funds are awarded, annual project status reports do not consistently map performance back to the original targets in the project applications. (As mentioned in the Limitations, the sub-activity notes that template changes were made in an effort to improve reporting. A consistent template has been implemented as of 2011).
  • Collaboration data: Data on collaborations and partnerships is difficult to quantify, as it is sometimes reported as the number of national/international partners and other times reported as the number of active formal collaborations/MOUs.
  • Annual reports: Annual reports summarize the information from the individual project status reports and are the means by which the sub-activity reports on the overall program performance. However, there are no summary statistics available on the number of projects that have resulted in partner take-up or the level of adoption of clean energy technologies. Only two of the five reports present data on performance indicators and output status. As a result, sub-activity management is unable to monitor the level of adoption and industry use of CESI technologies, and estimates of actual emissions reduction cannot be developed.
  • Financial data and project-level milestones: Complete and accurate financial data was difficult to obtain from sub-activity management and had to be compiled from various sources. Project-level reporting was highly variable, ranging from formal progress reports to informal notes summarizing telephone calls.

4.0 CONCLUSIONS AND RECOMMENDATIONS

A federally-funded, clean energy R&D initiative is an important part of the government’s ability to advance promising technologies and strengthen Canada’s technological and economic competitiveness. In this respect, the evaluation confirmed the relevance of the CESI sub-activity as it produced several positive results, and noteworthy project-level outcomes were observed. The evaluation found that CESI projects considerably expanded the general knowledge base on energy efficiency and emissions reduction, producing concepts, models and techniques, and publishing in peer-reviewed papers/conference proceedings, thus demonstrating the quality of this work. In addition, there are indications of improved efficiency among partners who have actually adopted CESI recommendations and are benefitting from increased energy savings.

It is commendable that a relatively small program with an annual budget around $5 million could help support meaningful partnerships between NRCan and multi-billion dollar industrial companies. Also noteworthy, the expertise of NRCan researchers consistently exceeded industry’s expectations, enhanced the Department’s reputation nationally and increased Canada’s visibility abroad.

The sub-activity currently aligns with federal clean air priorities and objectives to support economic growth and the competitiveness of Canadian industry. Despite this confirmed relevance and its many project-level successes, the evaluation raises fundamental questions about the sub-activity’s performance.
The sub-activity’s theory suggests that the knowledge gained and the technologies developed through the individual projects conducted will result in energy-efficient and emission-reducing technologies being adopted by industry more generally in the intermediate term. In the long-term this would then lead to industrial processes becoming more energy efficient and CAC and GHG reductions being achieved.

The sub-activity does not monitor industry adoption and the evaluation found limited evidence that knowledge gained from individual projects was being transferred to industry more generally. In short, the evaluation was unable to confirm a formalized transfer of project-level results to the program level or industry adoption of these project-level results. The lack of clear evidence of diffusion of results and take up by industry puts in question the eventual achievement of emissions reductions and economic gains.

S&T is a long-term process and the exploratory nature of CESI projects may contribute to the lack of evidence towards longer-term outcomes. Program management also believes that some diffusion of CESI results may happen through the efforts of NRCan’s Office of Energy Efficiency and the Innovation and Energy Technology Sector.

The evaluation makes four recommendations. Given the challenges the sub-activity has experienced in achieving its energy efficiency outcomes, the first recommendation follows evaluation findings that suggest a fundamental review of the program should be undertaken. The remaining three recommendations are meant to be taken under advisement in the review of the CESI program.

While the evaluation found that, in general, funding is sufficient to successfully carry out each project, the sub-activity is nonetheless attempting to reach to a broad industry audience with limited means. Reduced staff and budget over time, combined with two finite funding components and industry reluctance to invest in R&D, counteract the goal of long-term efficiency efforts across seven large industry sub-sectors. These factors, as well as the lack of a transfer strategy lead the evaluation to conclude than an overall review of the program is needed.

Recommendation 1: NRCan should review the future direction, design, and scope of CESI, and reassess whether it remains a priority and a viable program, given the need for the department to ensure the most effective use of its limited resources.

One way to align program scope with budget and actual resources would be to use the existing program tools (such as the Scope Tables) to identify one or two focused research areas or industry sub-sectors to prioritize. This may include a revision of the individual logic models of all funding components encompassed by the CESI sub-activity, to see what may be reasonably targeted within short, intermediate and long-term timeframes.

The sub-activity, as currently designed, has a low likelihood of producing impacts related to the intended outcome of industry adoption of new or improved CESI-developed technologies because it lacks a clear diffusion, outreach and knowledge transfer strategy. Consequently, CESI is unable to fully foster its central aims of industry-wide emissions/energy reductions, competitiveness, or improved industrial capacity through broad awareness.

Recommendation 2: An NRCan CESI program should include a sound strategy to disseminate the knowledge and transfer the technology gained on a project level more broadly across industry sub-sectors, thus facilitating industry-wide uptake and energy efficiency. Such a strategy needs to be developed and implemented collaboratively between NRCan’s Office of Energy Research and Development, Office of Energy Efficiency (OEE) and Innovation & Energy Technology Sector (IETS).

Some changes to ensure knowledge dissemination and technology transfer may include revising the current resource allocation structure to facilitate communication and outreach activities to increase the sub-activity’s effectiveness. This could also include more frequent program presentations and workshops, in which partner attendance would be encouraged to build strong awareness. As many of the CESI projects resulted in published peer-reviewed papers, these could be shared widely through an open-access repository. Similarly, software developed through CESI could be open-sourced to Canadian industry, enhancing market presence.

The fact that CESI supports one-to-one relationships instead of broad partnerships leads to another area for recommendation. The evaluation found that leveraging of partner funding was low and that there was little increase in new collaborations among industrial/institutional partners. In fact, as CESI did not systematically include a mandatory partnering requirement, the extent to which the NRCan researchers could truly engage their external counterparts was limited.

Recommendation 3: An NRCan CESI program should be designed to enhance researcher/partner interactions. As such, NRCan should: a) increase the diversity and number of external partners; b) develop specific criteria that will define the circumstances under which partners will not be required initially, as well as a process for determining when the engagement of partners will be required for these projects; and c) encourage the full contribution of other participating government departments.

One of the easiest ways to enhance researcher/partner interactions and leverage partner funds would be to make partnering compulsory in every CESI project. However, in certain circumstances, where the research is high-risk and long-term, this may not be possible. It is important for the Program to develop criteria to determine how these exceptional situations will be identified. For these projects, it is even more important that an engagement and dissemination strategy be developed and implemented to enhance take-up by industry of the results. This plan should also include annual assessments to determine whether the project has reached a sufficient level of maturity that the engagement of one or more industry partners should be required.

Similarly, as CESI is inter-departmental in nature, the continued participation of scientists and advisors from NRC and EC should be strongly encouraged. In addition, NRCan researchers can be a valuable asset to strategic planning and including them in the outreach process holds several advantages. For example, allowing them to meaningfully engage with outside organizations would: i) help diffuse of results and best practices gained through CESI research to a wider audience; ii) provide a way to gauge industry needs; and 3) determine which partnerships have the best potential for project success. NRCan could also consider allowing federal scientists to spend time at a partner institution and vice versa to enhance cross-fertilization of ideas.

In progressing towards outcomes, the evaluation also found that the sub-activity is not utilizing the most efficient or economic means available. An inadequate project selection process, lack of full-time management resources, and inconsistent measurement/monitoring of sub-activity impact, restrict the sub-activity from making sound evidence-based decisions. Thus, there is an opportunity to strengthen program implementation and communication, in order to improve decision-making and increase the overall efficiency and effectiveness of the CESI sub-activity.

Recommendation 4: An NRCan CESI program should include strong project selection processes and performance monitoring.

To reach the intended CESI objectives, strengthened program management may be achieved through identification of opportunities to improve the project selection process and the tracking and monitoring of impacts. For example, a first step may be to consolidate project data into a central database that would encompass research stage, financials, partner contacts, IP or competitive outputs, expected and achieved increase in economic competitiveness and in CAC/GHG reductions, etc. Aggregating and properly quantifying this information would improve performance monitoring, contributing to informed, evidence-based decision-making. In order to implement this suggestion, it would be necessary for the sub-activity to put in place a tracking mechanism that would monitor industry adoption of technologies for a period of time after the completion of the project.

Other suggestions would be to implement a formalized terms of reference for the project selection committee, increase involvement from other departments in the project selection process, and perhaps include an advisory role for an industry association. This would ensure a well-rounded selection committee with clear operating procedures, as well as the added benefits of incorporating industry needs into the selection of projects. Finally, increased feedback throughout the NRCan hierarchy, including a formal manager/researcher succession plan, would provide clarity behind decisions, ensure knowledge sustainability over time, and facilitate improved performance monitoring.

Finally, it is important to acknowledge that government targets and mechanisms for emissions reduction have changed significantly since the sub-activity’s inception. As such, and to maintain alignment with government and industry economic priorities, it is important that new projects emphasize the shorter term growth of economic and technological competitiveness by industry. CESI can help achieve this through increased efficiency which frequently leads to lower CAC/GHG emissions and this would, in turn, further increase Canadian industry competitiveness through the improvement of its image in international markets.

APPENDIX A PROJECT SELECTION FOR CASE STUDIES

Selection Criteria

At the time of selection, information was available for 48 projects (16 PERD; 20 CCTII; 12 ecoETI). Because of the low number of project reviews to be conducted and the need to encompass three different funding components, a multi-criteria selection approach was used. A sample of 19 potential projects was identified based on the maximization of the level of representation of the following variables:

  • Funding component: PERD, CCTII and ecoETI.
  • Research area: Process Integration and Heat Transfer (PERD); Advanced Combustion (PERD); Advanced Sensors and Controls (PERD); Transformative Technologies (PERD), Technology and Innovation GHG/energy (CCTII) and Emission Reduction (ecoETI).
  • Theme: Mining and Smelting, Petroleum Refining, Chemicals Production, Pulp and Paper, Cement and Lime, Iron and Steel, Integrated Application and Other Manufacturing.
  • Departments involved: NRCan (MMSL, CANMET), Environment Canada and National Research Council Canada.
    Size: Federal funding and in-kind contributions.
    Partnerships: Number of industrial and institutional partners.
  • Success/level of achievement: Projects were rated on their level of achievement towards short- to long-term outcomes. Information pertaining to achieved outcomes was found in the CESI annual reports and project status reports.

From this initial sample, five PERD-funded projects were selected, as well as two funded by CCTII and two by ecoETI. The final selection of projects (below) was provided to the Strategic Evaluation Division (SED) and sub-activity staff for approval during the design phase.

# Funding
Component
Research area Theme/ Sector Dep’t/ Agency Laboratory Size* Partners Level of
achievement
A1 PERD Process Integration and Heat Transfer Chemicals NRCan CANMET Energy
CETC Varennes
Med Industrial and Institutional
3 to 6
Intermediate,
possibly long-term
A3 PERD Process Integration and Heat Transfer Pulp and Paper NRCan CANMET Energy
CETC Varennes
Large Industrial and Institutional
3 to 6
Intermediate,
possibly long-term
B3 PERD Advanced Combustion Mining and Smelting NRCan CANMET Energy
CETC Ottawa
Med Industrial and Institutional
3 to 6
Intermediate
B4 PERD Advanced Combustion Mining and Smelting NRCan CANMET Energy
CETC Ottawa
Large Industrial and Institutional
more than 6
Intermediate,
possibly long-term
D4 PERD Transformative Technologies Mining and Smelting NRC Canadian Hydraulics Centre Large Industrial and Institutional
more than 6
Short term
IS4 CCTII Technology and Innovation
GHG/Energy
Iron and Steel NRCan CANMET Energy
CETC Ottawa
Med Industrial and institutional 1 to 2 Short term
MS2 CCTII Technology and Innovation
GHG/Energy
Mining and Smelting NRCan CANMET Energy
MMS-MMSL
Large Industrial
3 to 6
Intermediate,
possibly long-term
E10.007 ecoETI Emission Reduction Petroleum Refining NRCan CANMET Energy
CETC Ottawa
Small Industrial
1 to 2
Short term
E10.009 ecoETI Emission Reduction Petroleum Refining NRCan CANMET Energy
CETC Ottawa
Large Industrial
1 to 2
Short term

*Size: based on funding (cash and in-kind) in comparison to other projects under the same funding component

Interviews (n = 16+9)
Between August 19, 2011 and October 13, 2011, 25 interviews were conducted with the following types of stakeholders:

  • federal researchers (eight; one researcher interviewed for two projects);
  • representatives from industrial partner organizations (six);
  • Representatives from institutional partner organizations (one university and one non-for-profit research institute);
  • additional partners (nine)Footnote63.

Interviews were conducted by telephone in the official language of choice of the respondent (French or English). Respondents received an interview guide in advance. Interviews typically lasted 60 minutes (see distribution in Table 11). Interviews were recorded, and the recordings were deleted after completion of transcription and analysis. Interviews were not recorded when specifically requested by participants.

Table 12 Distribution of interviewed partners by project

#
Project Title Funding
Component
Theme/ Sector Number and Type of Interview Representative
Industry University Industry-based research institute or consortium
A1 Thermodynamically Guided Modeling and Integration of Separation processes PERD Chemicals 1
A3 Combined Energy and Water Optimization PERD Pulp and Paper 2 1
B3 Optimization Tools and Models for Industrial Combustion Processes PERD Mining and Smelting 1
B4 Efficient Industrial Combustion Furnaces and Processes PERD Mining and Smelting 3
D4 Optimization of Granular Multi-flow Processes PERD Mining and Smelting 1
IS4 Reduction of GHG Emissions in the Steel Industry CCTII Iron and Steel 1
MS2 Advanced Blasting from Comminution Process Optimization CCTII Mining and Smelting 2 1 2
E10.007 Sulphur Removal from Fuel Oils used in Heavy Industry ecoETI Petroleum Refining N/A*
E10.009 Development of Innovative Fuel Utilization Approaches to Reducing Energy/Emissions Intensity in Petroleum Refining Industry ecoETI Petroleum Refining 2

*Partner participation was limited strictly to the one-time provision of raw material. As such no partners were invited to participate in an interview.