Evaluation of the Built Environment Portfolio

Table of Contents

Executive Summary

1.0 Introduction

This report summarizes findings for the Evaluation of the Built Environment Portfolio (BEP), which—at the time of the evaluation—was co-managed by the Office of Energy Research and Development (OERD) within the Energy Sector and CanmetENERGY within the Innovation and Energy Technology Sector (IETS), Natural Resources Canada (NRCan). The Portfolio has undergone some governance and structural changes since 2011-12. For example, OERD moved to IETS in 2013.

The objective of the Built Environment Portfolio (BEP) is to generate new and improved knowledge, technologies and capabilities to increase energy efficiency and reduce air emissions in the built environment, focussing on the integration of leading-edge energy efficient and renewable energy technologies into new and existing housing, buildings and communities in Canada, including remote communities.

During the evaluation period, 2008-09 to 2011-12, the Built Environment Portfolio (BEP) consisted of one program (Buildings and Communities), 11 sub-programs, 76 R&D projects and $45.9M in NRCan expenditures. The 11 sub-programs focused on Integration R&D and Next Generation Technologies.Footnote 1 The BEP falls under Program Sub-Activity (2.1.3) – Clean Energy Science and Technology.

The evaluation covers both research and development (R&D) and demonstrations, but its focus is on R&D activities. It includes all science and technology (S&T) funding programs managed by OERD that contributed to the Built Environment Portfolio: Program of Energy Research and Development (PERD); ecoENERGY Technology Initiative (ecoETI); the Clean Energy Fund (CEF); and the ecoENERGY Innovation Initiative (ecoEII). PERD is the main funding source for the Portfolio and supports R&D activities. While the C-base (i.e., time limited) programs provide funding for R&D and demonstration activities, only a small portion of these funds were allocated to demonstration projects in this Portfolio.

Activities within this Portfolio include the development of scientific and technical knowledge for the advancement of codes, standards, regulations, and assessment tools; the development of next-generation energy efficiency and renewable energy technologies; and the strengthening of the capacity to integrate these technologies into new and existing houses, buildings and communities. Most of the research activities undertaken during the evaluation period were conducted in federal laboratories.

2.0 Evaluation Approach and Methodology

The evaluation examined the issues of relevance and performance and was conducted from 2012 to 2013. The evaluation was conducted in accordance with the 2009 Treasury Board Policy on Evaluation and the Departmental Strategic Evaluation Plan (2012-13 to 2016-17).

The evaluation used multiple lines of evidence to assess relevance and performance of the Portfolio: i) document review; ii) literature review; iii) interviews; and iv) case studies. A total of 112 interviews were conducted with NRCan, other federal departments and agencies and external stakeholders. Given the evaluation’s focus on uptake of knowledge and technology, the majority of interviews (66) were conducted with external partners and receptors. The external stakeholders consulted included a mix of real estate developers/builders, industry association members, manufacturers, universities, provincial and municipal representatives, utilities, building operators, engineers and architects. Twelve case studies were undertaken to assess the achievement of outcomes; factors that facilitate or hinder achievement of results; and best practices and lessons learned.

3.0 Evaluation Findings: Relevance

Overall, evaluation evidence indicates that BEP is relevant to environmental and economic needs and that the federal government and NRCan have a legitimate role in BEP R&D.

3.1 Continued Need for the Built Environment Portfolio

There is a continuing need for BEP R&D, particularly with respect to retrofits, to respond to technology and knowledge gaps for improving energy efficiency and reducing greenhouse gas (GHG) emissions in the built environment. Moreover, with its focus on energy efficiency, BEP RD&D has the potential to reduce strain on existing utility infrastructure by contributing to reduced energy use. As well, the Portfolio can provide increased economic benefits by building industry capacity; contributing to the harmonization of codes and standards; and facilitating progress towards commercialization of technologies by enhancing their technical feasibility and cost-effectiveness.

Evaluation evidence indicates that the Portfolio is generally aligned with identified needs related to enhanced capacity and competitiveness; an integrated approach to energy efficiency in the built environment; and information related to energy use and costs of energy efficient technologies or systems. However, external stakeholdersFootnote 2 indicate the need for more demonstration activities in the built environment to assess cost-effectiveness and practical feasibility of advanced technologies.

3.2 Alignment with Government Priorities and NRCan Strategic Objectives

BEP objectives are aligned with government priorities relating to clean energy, economic and northern development as identified in Speeches from the Throne and Budgets, and voluntary energy commitments to GHG reductions as set out in the Copenhagen Accord in 2009. BEP is also aligned with NRCan’s strategic objective relating to environmental responsibility. While BEP supports the mandates of other federal departments and agencies, there has been diminished involvement by some federal players (e.g., Environment Canada, Public Works and Government Services Canada, and Canada Mortgage and Housing Corporation) due to shifting priorities and decreased capacities. The Department of National Defence has increased its involvement in BEP for mobile and Northern applications.

3.3 Alignment with Federal Roles and Responsibilities

There is a legitimate role for the federal government and NRCan to conduct and support energy-related research activities in the built environment as expressed in NRCan’s policy of sustainable energy use, outlined in the Energy Efficiency Act. Evaluation findings indicate that the federal government also has a legitimate role to support the development of a knowledge base required for codes, standards and regulations.

As costs are a significant barrier to deployment of advanced technologies with potential to substantially reduce emissions, external stakeholders emphasize that government support is critical for the development of more cost-effective technologies and systems. In addition, industrial stakeholders emphasize that having government as a neutral knowledge disseminator and network facilitator is important given industry reluctance to share research results among competitors.

Federal and external interviewees generally agree that the federal role as a funder and direct R&D performer is appropriate given industry’s lack of capacity and the longstanding federal expertise in built environment research. The Canadian building sector comprises many small firms, most with limited to no capacity for longer-term R&D. There is overall agreement that a collaborative approach between government and industry is needed to both develop policy and build industry capacity.

The federal government role is perceived by most interviewees as appropriate for longer-term, pre-commercialization, and high-risk research. The federal role in supporting demonstrations is also perceived as important, particularly in the energy sector, due to high investment needs, low capital turnover, competition from existing, mature technologies, and high capital costs.

4.0 Evaluation Findings: Performance

Overall, the Portfolio has produced most of its planned outputs and has made good progress on immediate and intermediate outcomes. As most of BEP research is applied, it is too early to show significant progress regarding longer-term environmental and economic impacts for much of the Portfolio’s expenditures. Moreover, case study and document evidence indicate that lengthy time horizons can be required for realization of these outcomes. As well, achievements in the built environment are constrained by low natural gas prices, and the complexity of the buildings and communities sectors. Nevertheless, there is evidence to indicate progress towards longer term outcomes.

4.1 Effectiveness

Production of Outputs

According to BEP internal documentation, 459 publications/reports have been disseminated through the following means during the evaluation timeframe:

  • 72 refereed journal articles;
  • 80 non-rfereed journal articles;
  • 157 conference proceedings/papers;
  • 117 internal technical reports; and
  • 33 client reports.

In addition, an analysis of program documentation identified the following outputs produced during the timeframe of the evaluation: 60 guidance documents (e.g., technology assessments, plans and specifications, design guidelines, best practice guides); 28 enhanced models, simulations or software (e.g., Spatial Community Energy Carbon and Cost, Regional Energy Analysis Model, Hot3000) for use by researchers and end-users; 27 data files; 15 large-scale prototypes; 53 field trials; 7 processes or methodologies; and 6 designs or construction specifications.

Most outputs (70%) were delivered as planned. Internal interviewees indicated that this was a reasonable achievement given the unpredictable nature of R&D. The most commonly cited reasons for project delays were problems with the testing or demonstration sites and difficulties retaining external partners. Loss of capacity in some government departments was cited as a reason for delays in some cases. External stakeholders generally considered the R&D outputs to be of high quality.

Contribution to National Codes, Standards and Energy Efficiency Programs

BEP has provided important technical input into the development of codes, standards and regulations. The Portfolio has contributed to the National Energy Code for Buildings and to the development of five standards for the Canadian Standards Association. BEP has made valued contributions to NRCan’s Office of Energy Efficiency programming, particularly with respect to housing and equipment.

Influence on Capacity of External Stakeholders

Evidence indicates that external partners have enhanced their awareness, knowledge and skills of the built environment sector as a result of involvement in Portfolio projects. There is likely broader capacity enhancement among some end-user groups (e.g., residential home builders) facilitated by strong linkages with industry associations in housing and renewables. Strengthened dissemination would better optimize the achievement of this outcome.

Influence on Use and Implementation of BEP Knowledge and Technologies

Evaluation evidence shows that there is good uptake of BEP knowledge among external project partners consulted. The majority of external partners interviewed indicated that they were using BEP knowledge in a variety of ways. Residential builders reported increased use of energy efficient building design and construction practices. Industry stakeholders generally indicated that they used BEP project involvement to promote and market their companies. Academic stakeholders consulted reported that BEP research facilitated their decisions as to future research directions. Many external researchers also highlighted that BEP projects gave them access to databases and monitoring files which were critical sources of information upon which to build their research. Manufacturers consulted indicated that they developed more energy efficient products as a result of their involvement in the Portfolio.

Influence on Use and Implementation by Sub-Program

Assessment of the levels of uptake with respect to housing, buildingsFootnote 3, communities, and next generation technologies shows some variation. This is related to a number of factors including the stage of research; and the different opportunities and challenges facing housing, building and communities’ research projects. Progress towards use and implementation of BEP knowledge and technology is summarized below by sub-program.

  • For the Net-Zero Energy Housing (NZEH) sub-program there has been good uptake particularly by early adopters (builders) with potential for broader deployment of energy efficient housing design, technology and construction practices. For example, builders involved in the EQuilibrium Housing Initiative are building more energy efficient homes based on the experience gained from the Initiative. Another notable example is the builder-driven technology assessment process developed under the Local Energy Efficiency Partnerships (LEEP) project which helped to accelerate the market adoption of energy efficient innovations for near and net-zero housing. The effectiveness of this sub-program has been influenced by the strong linkages between CanmetENERGY and the Office of Energy Efficiency. In addition, the sub-program has well established relationships with national housing associations. The Canadian Centre for Housing Technologies (CCHT) provides the infrastructure for testing energy efficient housing technologies and facilitates strong linkages among NRCan, National Research Council (NRC); and Canada Mortgage and Housing Corporation (CMHC).
  • The Net-Zero Energy Building (NZEB) sub-program made important contributions to the National Energy Building Code through the development of 16 building archetypes and the provision of expertise. CanmetENERGY also provided design expertise and support for the construction of NRCan’s Materials Technology Laboratory (MTL)Footnote 4, a Leadership in Energy and Environmental Design (LEED) platinum building. This sub-program also generated useful information for green building policy through the Post Occupancy Evaluation of Green Buildings Project. Some work in this sub-program regarding the development of design guidelines and energy use profiles for energy efficient multi-residential buildings experienced challenges in sustaining partnerships due to capacity issues among some external partners. While there are indications of good progress in this sub-program, its effectiveness could be enhanced through strengthened linkages with other BEP research and relevant associations.
  • The Community Pinch sub-program has developed and piloted integrated energy mapping tools in British Columbia and Ontario municipalities, contributing to a more consistent method for characterizing energy use and emissions in the building stock. Factors influencing effectiveness of this sub-program include provincial policies pertaining to energy conservation and climate impact mitigation strategies and the complexity of community planning and governance processes. While this sub-program has good linkages with appropriate national level associations (for example the Canadian Federation of Municipalities), communications with these organizations could be further strengthened.
  • The Next Generation Technology sub-programs have advanced the performance (e.g., energy efficiency) of a number of technologies, such as residential micro-cogeneration systems; high performance wall systems; ejectors/heat pumping systems; solar canopy system; microblinds; controllability of LED lighting; and solar-driven liquid desiccant air cooling systems. Some projects within the sub-programs did not sufficiently progress largely due to challenges in securing field test sites, sustaining external partners, loss of capacity of other government departments, equipment issues, or laboratory repairs.
Economic and Environmental Impacts

During the timeframe of this evaluation there have been four products/systems that have been commercialized: Diagnostic Agent for Building Operation (DABO) software; zoned comfort system; solar canopy system; and advanced phase change material. In the commercial sector, R&D work on a carbon dioxide secondary loop refrigerant has led to a new Loblaw’s Superstore successfully using the technology and Sobeys has adopted CO2 as the sole refrigerant for all of its new and major retrofit stores. In addition, many industry partners noted that their involvement in BEP increased their capacity and visibility, which in turn facilitated the companies’ marketing and promotion efforts, leading to an increased demand for their work.

There is an argument to be made that BEP research activities have the potential to indirectly contribute to increased energy efficiency and GHG emission reductions through its longstanding technical input into codes, standards and energy efficiency programs. While the evaluation cannot attribute longer-term impacts solely to BEP, Canadian environmental trends in the residential and commercial sectors indicate some improvements in energy efficiency and energy intensity from 2007 to 2010.

Factors Influencing Effectiveness

The consistency of funding as provided through PERD was cited as a key positive factor by interviewees and supported through case study evidence. The sustained funding has ensured continuity in research activities, developed and retained research capacity, and facilitated linkages with external stakeholders.

Internal NRCan interviewees frequently noted that the continuing decline of NRCan A-base (i.e., ongoing) funds is constraining the effectiveness of the Built Environment Portfolio by limiting collaboration among federal researchers and the capacity to leverage PERD funds. There is a general perception that declining A-base funds increase pressures on PERD processes such as project selection.

Both internal and external interviewees note that insufficient funding targeting demonstration and deployment activities in the built environment further constrain technology transfer and uptake. Moreover, some interviewees noted that there are inadequate funding mechanisms (e.g., contribution agreements) for federal and private sector collaboration on demonstration projects. The literature review on best practices in energy research supports the need for collaborative public and private sector demonstration projects as an effective means of enabling technology transfer.

There are a number of market barriers to broader deployment of energy efficient technologies in buildings and houses. The relatively low price of conventional energy sources, especially natural gas, make alternative energy technologies less economically feasible, reducing economic incentives to be energy efficient. In addition, high up-front capital costs of new construction or retrofitting is a key barrier to building net-zero energy homes and buildings. In the building sector "split incentives" between building owners and tenants is a major barrier to widespread deployment of energy efficient systems and technologies: Building owners may not always benefit from an efficiency project as they can often pass energy costs on to their tenants.

4.2 Efficiency and Economy

There are indications that BEP is an efficient and economic portfolio, but there are opportunities for further enhancements. Measures of efficient and economic practices were found with respect to adequate leveraging of resources; appropriate broad-spectrum design as supported by interview evidence and identified best practices for publicly funded R&D; and improved planning and management practices in comparison to practices during the previous program cycle (2003-04 to 2008-09). The evaluation identified areas for improvement related to internal communications and coordination, project selection and review, performance measurement, and dissemination.

Portfolio Design, Planning and Management Practices

The broad-spectrum BEP was viewed by most internal interviewees as appropriate to stimulate a wide range of promising technologies and to accommodate an integrated approach to housing, buildings and communities. Nonetheless, the evaluation concluded that some streamlining of projects and sub-programsFootnote 5 may be required to maximize impact.

The Strategic Planning Process for the 2008-09 to 2011-12 cycle was generally viewed as an improvement to previous efforts in that it better linked priorities to the Portfolio framework. Another noted improvement was the inclusion of NRCan’s Office of Energy Efficiency in the Built Environment Portfolio Committee in 2010-11. However, some federal interviewees indicated that planning requires further enhancements, such as more systematic input of external stakeholders.

Two independent performance reviews were conducted (mid-cycle and end-of-cycle) to assess the Portfolio’s progress. The results of the end-of-cycle review were linked to the most recent planning cycle that commenced in 2012-13, an indication of good management.

Leveraging of Funds and In-kind Resources

Between 2008-09 and 2011-12, the BEP (includes NRCan and other federal government BEP expenditures) expended $60.7M and leveraged $33.5M in cash and in-kind contributions from non-Government of Canada sources including provinces, industry, academia, and non-governmental and international organizations. The ratio of Government of Canada funds to non-Government of Canada contributions was 1: 0.55.

During the same time period, the Portfolio received $38.8M from PERD, ecoETI, CEF and ecoEII and leveraged $55.4M in cash and in-kind contributions from other sources (i.e., NRCan A-base, other government departments, and non-Government of Canada sources). The ratio of BEP funds to all other funds and in-kind sources was 1: 1.4.

Compared to the other funding envelopes, PERD projects had the highest leveraging ratio. During the evaluation timeframe, total PERD funding was $21.7M compared to $47M in cash and in-kind contributions from other sources (i.e., NRCan A-base, other government departments, provinces, industry, academia, and non-governmental and international organizations). The ratio of PERD funds to all other sources was 1: 2.2.

These leveraging ratios are similar to those of other NRCan R&D Portfolios (e.g., Clean Transportation Systems). However, the leveraging ratio of BEP funding to other funding is slightly lower as compared to the built environment R&D conducted in the previous cycle (i.e., 2003-04 to 2008-09).

Opportunities for Improvement

Project Selection. NRCan interviewees were generally satisfied with the project selection process for C-base funding programs (i.e., CEF, ecoETI, ecoEII). With respect to the project selection process for the ongoing PERD, it was noted that it had undergone changes during the 2008-09 planning cycle with a view to enhancing collaboration among federal researchers. However, there is a perception among federal interviewees that the project selection process could be further enhanced by ensuring closer alignment with science-based criteria and priorities.

Project Review. Current measures to review progress include mid-year sub-program meetings, annual workshops and Portfolio Committee meetings, end-of-cycle external review, and discussion and recommendations of the Portfolio Committee. Internal interviewees cited the need for further improvements to the project review process to clearly identify areas of high and low performance. Specific criteria could be established to further facilitate decisions as to whether to continue project funding.

Internal Communication and Collaboration. In the evaluation period, there have been increased efforts to improve collaboration and communication among researchers. For example, researchers were required to submit integrated proposals for each sub-program during the 2008-09 and the most recent PERD planning cycles. While the evaluation found some examples of good internal communication and collaboration, particularly within some sub-programs, there is a need to further strengthen collaboration. There is also a need for greater coordination of planning activities between OEE and BEP. Improved communications between OEE Buildings and BEP is also recommended.

Performance Measurement and Reporting. BEP tracks its outputs and finances through project reporting templates, which are aggregated into Program Annual Reports (PARs). However, the format of the reporting templates does not lend itself to consistent reportingFootnote 6 , nor to outcome reporting. More consistent and systematic reporting would facilitate comprehensive semi-annual and annual reporting at the sub-program and portfolio levels and identification of optimal and sub-optimal performance areas. There is also the need to review performance metrics directed towards the community sub-program to facilitate clearer reporting of results.

Dissemination of Results. While the evaluation found evidence of some good dissemination practices, the absence of a comprehensive dissemination strategy was a significant impediment to effectiveness. Key barriers to the development and implementation of a strategy included insufficient resources, travel restrictions, limitations of publishing on the federal government and NRCan websites in terms of space, formatting, and accessibility, length of time for approval to publish results, and insufficient clarity as to responsibilities for dissemination.

5.0 Conclusions, Recommendations and Management Responses

Overall, BEP programming is relevant and effective and should continue with some modifications to its design and delivery. The Portfolio activities are consistent with federal priorities and the federal and NRCan roles. BEP programming is needed to advance built environment technologies and knowledge that have the potential to contribute to economic competitiveness and GHG emissions reductions and increased energy efficiency.

The Portfolio has performed well in developing knowledge and tools that have made important technical contributions to codes, standards and energy efficiency programs. BEP has largely been effective in achieving immediate and some intermediate outcomes, although progress towards longer-term outcomes is inconsistent across sub-programs. Achievement of intermediate outcomes relating to use and deployment of research outputs is more evident among the Net Zero Energy Housing and several next generation technology areas. While the inconsistencies in uptake relate in large part to external factors, further enhancements to internal communication and collaboration would facilitate more consistent outcome achievement. A more comprehensive dissemination strategy is required to enhance the effectiveness of the Portfolio as a whole. Insufficient resources for built environment demonstrations and deployment activities were viewed as key factors impeding innovation. Declining resources, in particular the declining NRCan A-base for research, was a common concern cited among federal interviewees as impeding the effectiveness of BEP.

The Evaluation has four recommendations to enhance the design and delivery of the Portfolio.

Recommendations and Management Responses

Recommendations Management Response Responsible Official/Sector (Target Date)
Recommendation 1: NRCan continue to identify and strengthen strategies to improve internal communication and coordination among BEP researchers and between BEP and OEE, particularly on the buildings side. ACCEPTED: IETS-OERD currently requires PERD program coordinators to organize an annual workshop in which all BEP researchers are invited to participate, present their work, network and discuss opportunities to collaborate. Furthermore, sub-program teams are required to meet at least once per year (at mid-year), and many teams meet on a more frequent basis. IETS-OERD will explore further opportunities for more frequent communication among researchers.

Senior OEE technical representatives from both Housing and Building Divisions were invited to join the Built Environment Portfolio Committee in 2011, during the last funding cycle. OEE representatives provided input on project selection and also participated in the annual workshop, and this will continue. In addition, IETS-OERD will meet to discuss opportunities to create linkages between BEP research and OEE’s programs, particularly on the buildings side.
ADM IETS

March 2014
Recommendation 2: NRCan continue to strengthen its current review process to clearly identify which BEP projects are not progressing as planned, and to take appropriate actions. ACCEPTED: IETS-OERD has strengthened the review process. The year-end Portfolio workshop and mid-year reviews for all sub-program areas are now compulsory. Several measures for identifying projects that are not progressing as planned have been implemented in the current PERD cycle (2012-16). The progress reporting templates now include fields to report percent completion of planned work, status of tasks (on schedule, delayed, etc.), reasons for delays, and corrective measures that will be taken to address delays and issues that can affect project delivery. Completed progress reports are closely reviewed and issues are flagged by IETS-OERD for appropriate action. ADM IETS

Completed
Recommendation 3: NRCan enhance dissemination of BEP outputs by developing and implementing a more comprehensive dissemination strategy that includes strengthened links to industry associations. ACCEPTED: IETS-OERD currently requires researchers to include a dissemination plan for each project proposal submitted for funding and requires that they report on these activities annually. Such activities include peer-reviewed papers published in professional journals and presentations at conferences, which are accessible to industry and policy-makers.

As part of its commitment to improving dissemination of outputs in all Portfolios, IETS-OERD is exploring, with NRCan corporate sectors, opportunities to broaden further the reach of the knowledge products arising from its funding programs, and will build these into its Portfolio action plans.
ADM IETS

March 31, 2015
Recommendation 4: NRCan further enhance the BEP performance measurement and reporting strategy by improving the consistency of reporting and making clear links to outcomes. Reporting on outcomes would be facilitated by regularly obtaining feedback from external partners and receptors. ACCEPTED: To further improve the consistency of reporting, IETS-OERD 1) has implemented the standard reporting template in the current funding cycle; 2) has instituted a process whereby all submitted reports are reviewed for completeness and quality, and are returned to the project leader for further work, if necessary; and 3) held a workshop on November 7, 2013, with BEP project leaders to share best practices for completion of the reporting templates and improving the consistency of reporting.

Reporting on outcomes is further facilitated by obtaining feedback from external partners. This is achieved through annual reporting on outcomes, which is required of all recipients of contribution agreements for a period of five years after the end of their project.
ADM IETS

Ongoing

1.0 Introduction and Background

1.1 Introduction

This report summarizes evaluation findings for the Evaluation of the Built Environment Portfolio (BEP), which is co-managed by the Office of Energy Research and Development (OERD), and CanmetENERGY, Innovation and Energy Technology (IETS), Natural Resources Canada (NRCan). In 2013, OERD moved from the Energy Sector to the IETS. The structure of the Portfolio has undergone some changes since 2011-12. BEP is currently the Built Environment Technology Area within the End-Use Portfolio.

OERD was located within the Energy Sector for the duration of the evaluation period and moved from the Energy Sector to the IETS in 2013.

The evaluation examined the issues of relevance and performance and covered the period from 2008-09 to 2011-12. It covers all 11 sub-programs, and 76 R&D projects under the Portfolio and $45.9M in NRCan expenditures. The evaluation was conducted in accordance with the 2009 Treasury Board Policy on Evaluation and the Departmental Strategic Evaluation Plan (2012-13 to 2016-17).

1.2 Overview of the Built Environment Portfolio

The objective of the Built Environment Portfolio (BEP) is to generate new and improved knowledge, technologies and capabilities to increase energy efficiency and reduce air emissions in the built environment, focussing on the integration of leading edge energy efficient and renewable energy technologies into new and existing housing, buildings and communities in Canada, including remote communities.Footnote 7

The Built Environment Portfolio consists of one program (Buildings and Communities), 11 sub-programs and 76 R&D projects.Footnote 8 Projects received funding from one or more of the following four federal S&T funding programs managed by OERD: Program of Energy Research and Development (PERD); ecoEnergy Technology Initiative (ecoETI); the Clean Energy Fund (CEF); and the ecoEnergy Innovation Initiative (ecoEII).

The objective of PERD, a long-standing fund, is “to fund research and development designed to ensure a sustainable energy future for Canada in the best interests of its economy and the environment”.Footnote 9 PERD priorities are periodically re-aligned to balance current government priorities and longer-term needs with the aim of ensuring that, as an ongoing core program, it provides the science and technology foundation upon which sun-setting energy RD&D programs can be built.

The ecoETI funds “research, development and demonstration activities to support the development of the next generation energy technologies needed to achieve emissions-free energy production and use, as well as to generate the new scientific knowledge essential to support the proposed regulatory approach to the Clean Air Agenda, including the scientific and technical background for establishing meaningful codes and standards.” Footnote 10

The objective of the CEF is to “support the development and demonstration of the new, cutting-edge energy technologies that are needed to reduce greenhouse gas and other air emissions in energy production, transmission, distribution and use.” Footnote 11

The objective of the ecoEII fund is to support energy technology innovation to produce and use energy in a cleaner and more efficient way. Footnote 12 The Initiative addresses the need to reduce the risk and cost of new, clean energy technologies as the current suite of clean energy technologies are neither sufficiently developed nor ready for mass market deployment.

Activities within this Portfolio include the development of scientific and technical knowledge for the advancement of codes, standards, regulations, and assessment tools; the development of next-generation energy efficiency and renewable energy technologies; and strengthening the capacity to integrate these technologies into new and existing houses, buildings and communities. Footnote 13

1.2.1 The PAA for the Built Environment Portfolio

The Built Environment Portfolio falls under the Clean Energy Systems for Buildings and Communities Strategic Priority Area, one of six strategic priority areas that receive funds from NRCan’s Office of Energy Research and Development for energy research, development and technology demonstration (RD&D) activities. Footnote 14

The position of the BEP within NRCan’s 2011-12 Program AlignmentArchitecture (PAA) is shown in Table 1.

Table 1: Position of Built Environment Portfolio within NRCan’s 2011-12 PAA
PAA Unit Expected Results/ObjectiveFootnote 15
Strategic Outcome (2)

Environmental Responsibility
Canada is a world leader on environmental responsibility in the development and use of natural resources
Program Activity (2.1)

Clean Energy
Increased energy efficiency, increased production of low-emission energy, and reduced environmental impacts associated with energy production and use
Program Sub-Activity (2.1.3)

Clean Energy Science and Technology
The advancement of clean energy knowledge and technologies that address the needs of partners and stakeholders.
Strategic Priority Area 3

Clean Energy Systems for Buildings and Communities
To reduce the overall energy intensity of Canada’s buildings and community energy systems and consequently, their associated greenhouse gases (GHG) and criteria air contaminants (CAC) emissions while, at the same time, providing Canadian companies with potential economic opportunities.

1.2.2 Expected Results and Theory of Change

A logic model for the BEP was developed based on a document review and consultation with the Advisory Committee for this evaluation study. The logic model is consistent with those found in the Strategic and Program Plans for the Portfolio and for the relevant funding programs. As well, theories of change from the literature relating to R&D and the innovation spectrum were used as guidance in developing the model.

BEP Logic Model

BEP is a set of inter-departmental co-ordinated research activities that aim to advance the development, integration and use of technologies to increase energy efficiency and reduce air emissions in new and existing housing, buildings and communities.Footnote 16 While the BEP involves some basic research, most activities can be classified as applied research.Footnote 17 As depicted in the logic model, the BEP sub-programs are expected to influence complementary immediate outcomes (e.g., more informed decision-making of policy makers; increased knowledge and skills of the built environment sector) leading to reduced market barriers and increased use of Near- and Net-Zero tools, knowledge, technology and solutions. This in turn is expected to contribute to more widespread deployment of energy efficient built environment solutions which link to environmental and economic benefits. When considering the success of an R&D program it is important to consider where projects and activities are located on the innovation continuum as this affects the choice of metrics used to measure their performance.

Immediate Outcomes

The three immediate outcomes are:

  • More informed decisions of policy makers, regulators, standards committees, and governments regarding the development of policy, codes, standards, regulations
  • Increased capacity of the private sector, governments, and academia to further advance R&D, promote, and deliver training in Near-Zero and Net-Zero energy built environment solutions
  • Enhanced awareness, knowledge, skills, and technologies of the built environment sector (industry, planners, utilities, and decision-makers, governments) to make decisions, plan, design, construct, operate and measure Near-Zero and Net-Zero houses, buildings and communities
Intermediate Outcomes

The two intermediate outcomes are:

  • Reduced market barriers (e.g., new and revised codes, standards, regulations and test methods for the use and integration of Near-Zero and Net-Zero energy built environment solutions; development of more cost-effective technologies)
  • Increased use, implementation and integration of Near-Zero and Net-Zero energy built environment knowledge, tools, technologies and solutions
Ultimate Outcomes

The two ultimate outcomes are:

  • Near-Zero and Net-Zero energy built environment solutions are pervasive and common practice across Canada thereby reducing the energy intensity of Canada’s buildings and community energy systems and consequently, their associated greenhouse gases (GHG) and criteria air contaminants (CAC) emissions as well as creating economic opportunities for Canadian companies; and
  • Policies, codes, standards and regulations are supportive of Near-Zero and Net-Zero energy solutions in the built environment.
Theory of Change
Figure 1: Innovation Continuum

Figure 1  Innovation Continuum

Text version

Figure 1: Innovation Continuum

This simple graphic, Figure 1, depicts the stages of the innovation continuum: Research, Development, Demonstration and Deployment.

The technology innovation cycle can be divided into four stages: Research, Development, Demonstration, and Deployment (RDD&D).Footnote 18 Although BEP primarily involves R&D activities, progression to widespread deployment is complex and requires good linkages to demonstration and deployment. While the innovation continuum depicted in Figure 1 is linear, it should be noted that the path to widespread adoption and commercialization is complex and involves numerous activities in addition to research and development such as business planning, identifying customer needs, and market transformation activities.Footnote 19 Feedback from the market and from technology users can drive further RD&D.Footnote 20 Strong linkages between market experience and further technical development is especially important.Footnote 21 Assessment of R&D impacts in the buildings sector must also consider factors such as economic conditions, market events, and costs of energy and building materials.

There is a need for an integrated approach between RD&D and deployment activities to facilitate transfer of technologies. Technology deployment in the marketplace in conjunction with R&D efforts is a key element to accelerate change, although it can be very difficult to distinguish between the effects of R&D and market deployment.Footnote 22

Analysis of performance, especially with respect to increased energy efficiency, should also consider the “rebound effect” and the “leakage effect”. The rebound effect occurs when technical changes reduce the costs of an activity which leads to an increase in the activity. Therefore, it is difficult to know if demand for energy will increase or decrease as a result of energy efficiency improvements. The leakage effect occurs when energy savings are used to increase energy use in other areas. While rebound and leakage effects will erode some of the energy savings from energy efficiency programs, most of savings are likely to remain. Moreover, accelerated improvements in minimum standards and codes can reduce or eliminate the rebound effect.Footnote 23

1.2.3 Delivery Approach & Governance Structure

The BEP receives most of its funding from programs managed by the Office of Energy Research and Development (OERD), Innovation and Energy Technology Sector (IETS)Footnote 24 (i.e., PERD, ecoETI, CEF and ecoEII funding envelopes). The BEP is comprised of eleven sub-programs and is co-managed by OERD and CanmetENERGY, IETS.Footnote 25

The R&D projects within these sub-programs are delivered by federal organizations via MOUs and non-federal delivery agents (e.g., industry; universities; etc.) through contributions. The CanmetENERGY Laboratory, within IETS, is a key delivery agent for the Portfolio.

As depicted in Figure 2, the OERD governance structure includes a Panel of Assistant Deputy Ministers on Energy Science and Technology (S&T), a Director General Committee on Energy S&T, and the BEP Committee.Footnote 26 The Portfolio committee receives advice on energy S&T priorities from several sources, including external stakeholders.

Figure 2: Built Environment Portfolio Governance Structure

Figure 2 Built Environment Portfolio Governance Structure

Text version

Figure 2: Built Environment Portfolio Governance Structure

Figure 2 is an organizational chart which shows the governance structure of the Built Environment Portfolio. The Portfolio is overseen by the Assistant Deputy Minister (ADM) Panel on Energy Science and Technology. The Director General Committee, Chaired by the Director General of Office of Energy Research and Development (OERD), provides advice. Just below the ADM panel is the Built Environment Portfolio Committee which is Chaired by an OERD representative. The Program Lead is a representative from the Innovation and Energy Technology (IETS) Sector. An OERD S&T Advisor provides guides to the Portfolio Committee. Beneath the BEP Committee are the 11 Sub-Programs with each Sub-Program having a sub-program coordinator. Sub-program coordinators can be representatives of IETS or other government departments receiving PERD funding. Below the 11 sub-programs are the projects with each project being led by a project manager.

OERD responsibilities include managing funding operations, meeting accountability requirements (including preparing Portfolio and Program Annual Reports), providing expert advice and support to Portfolios, providing secretariat support to the ADM Panel and DG Committee, networking and outreach.Footnote 27

The Assistant Deputy Minister (ADM) Panel on Energy S&TFootnote 28 is responsible for providing strategic advice on federal energy S&T to OERD, reviewing and approving Portfolio Strategic Plans, and advising on the distribution of effort across the innovation spectrum in each Portfolio (i.e., approving suites of projects).Footnote 29

The Director General (DG) Committee on Energy S&T, a horizontal committee, is responsible for considering and representing the operational level views and interests of its member departments or agencies to OERD, as well as to the ADM Panel on Energy S&T. It also reviews annual energy S&T allocation proposals from OERD.Footnote 30 The Chair of DG Committee on Energy S&T is a non-voting member of the ADM Panel and serves as the liaison between the two groups.

The Built Environment Portfolio Committee, in conjunction with the other portfolio committees, is responsible for managing the planning and administration of PERDFootnote 31 and other funding programs. The BEP Committee is also responsible for developing and implementing the Portfolio’s strategic plans and reviewing progress against their milestones. The Committee is responsible for project selection. The Portfolio Committee is chaired by an OERD Energy S&T Programs Assistant Program Director, and includes the Program lead and the OERD S&T Advisor.Footnote 32 In 2010-11, the membership of the Built Environment Portfolio Committee was expanded to include not only representatives of the five Science-based Departments and Agencies (SBDAs) participating in Portfolio activities, but also representatives from several NRCan Divisions/organizations with an interest in energy issues in the housing, building and communities sectors. Other departments are also present on the Committee in an advisory capacity, which is intended to increase take-up of results.Footnote 33

The Program lead is appointed by OERD and manages the Portfolio. In the case of BEP, the program lead is an employee of CanmetENERGY, IETS. The program lead produces the Strategic Plan and the Program Plan in collaboration with the OERD S&T Advisor and Portfolio committee members; assembles interdepartmental/agency teams to prepare proposals at the sub-program level for projects that contribute to Portfolio-level outcomes; recommends and manages the allocation of resources to achieve the outcomes established in the Program Plan; organises an annual program progress review meeting in collaboration with OERD and the Portfolio Committee; and reports on results through the Program Annual Report (PAR).Footnote 34

Sub-program co-ordinators organize mid-year progress review meetings; plan and manage matters at the sub-program level (e.g., bringing the team together); identify the number and scale of projects to achieve sub-program objectives and maximize synergies; and report on progress and financial matters at the sub-program level.Footnote 35

Recipient departments/agencies perform the R&D activities to which they have committed in the program plan, participate as required in working groups, committees, international energy S&T activities, conduct annual reviews of progress in the program areas that they lead, cooperate with OERD in facilitating the dissemination of S&T results, cooperate with audit and evaluation requirements, take a results based approach to managing their programs, and submit financial reports and annual and final project reports.Footnote 36

1.2.4 Resources at the Sub-sub Activity Level

As shown in Table 2, total NRCan expenditures from 2008-09 to 2011-12 were about $45.9 million which includes $21.7M from Program of Energy Research and Development (PERD); $7.1M from A-base and non A-base; $7.0M from ecoEnergy Technology Initiative (ecoETI); $7.0M from the Clean Energy Fund (CEF); and $3.0M from the ecoEnergy Innovation Initiative (ecoEII).

Table 2: NRCan Expenditures - Buildings and Communities 2008-09 to 2011-12 (in $000s)
Funding Sources 2008-09 2009-10 2010-11 2011-12 Total
PERD $5,135 $5,345 $5,320 $5,921 $21,721
ecoETI $2,672 $2,352 $2,009 n/a $7,033
CEF n/a n/a $2,194 $4,820 $7,014
ecoEII n/a n/a n/a $3,037 $3,037
NRCan A-base $604 $615 $913 $1,083 $3,215
NRCan other* $1,049 $1,025 $1,086 $753 $3,912
Total NRCan $9,460 $9,337 $11,522 $15,614 $45,932

* Includes cash and in-kind, from OEE – Housing; CanmetENERGY; and OERD (e.g., to conduct internal reviews). Source: OERD funding numbers for PERD, ecoETI, CEF and ecoEII provided by OERD. Other federal and non-federal contributions taken from 2011-2012 Year End Reports.

Table 3 depicts the timelines of the key funding sources for this Portfolio.

Table 3: Timelines of funding sources, amounts and # of projects funded
2008-09 2009-10 2010-11 2011-12
PERD ($21M) $21.7M; 22 projects
ecoETI $7M; 11 projects  
CEF     $7M; 15 projects
ecoEII       $3M; 28 internal track-A projects

During the timeframe of the evaluation there was an average of 57 FTEs/year. The number of FTEs increased in 2010-11 and 2011-12 reflecting the commencement of the CEF and ecoEII funding. In addition, a new sub-program, Distributed Energy Resources, became part of the Portfolio in 2011-12.

Total BEP funding from all sources for the evaluation period was about $94.2M. Figure 3 below shows the distribution of BEP funding by source. NRCan contributed $45.9M while $48.2M in cash and in-kind were contributed by other federal sources, industry, provinces/territories, international institutions, academic institutions and NGOs.

Figure 3: Total BEP Funding from 2008-09 to 2011-12

Figure 3: Total BEP Funding from 2008-09 to 2011-12

Text version

Figure 3: Total BEP Funding from 2008-09 to 2011-12

Figure 3 is a pie chart showing total BEP funding from 2008-09 to 2011-12. The largest source of funding comes from NRCan ($45.9M). The following sources of funding are: Industry $19.1M; Other federal ($14.7M); Provincial/Territorial/Municipal ($6.7M); Academia ($3.7); and Non-Governmental Organizations ($.13M).

1.2.5 Sub-Program Descriptions

The eleven sub-programs within the Built Environment Portfolio are a set of inter-departmental coordinated research activities that aim to advance the development, integration and use of technologies to increase energy efficiency and reduce air emissions in new and existing housing, buildings and communities. Sub-programs are groups of interrelated or complementing S&T projects that are collectively designed to meet established short- to mid-term goals within a specific time frame.Footnote 37

The sub-programs are described briefly in Table 4.

Table 4 : Components of the Clean Energy Systems for Buildings and Communities Strategic Priority Area and Expenditures
Sub-Programs Description NRCan Expenditures* and Timeline
Integration R&D Sub-programs

1. Net-Zero Energy Housing (NZEH)

Focused on integrated research and deployment of energy efficient and renewable source technologies aimed at achieving net-zero energy use in new and existing housing. The R&D aims to facilitate energy efficiency technology integration in the housing market. $4.6M
2008-09 to 2011-12

2. Net-Zero Energy Buildings (NZEB)

Aims to advance net-zero building design by establishing a range of feasible solutions, developing new design methods and tools and assessing the possibilities of enhancing green building rating and design approaches. R&D will facilitate the development of Net-Zero energy Design tools; support the development of a new regulator platform and net-zero energy applications. $6.2M
2008-09 to 2011-12

3. Community PINCH Technologies (methodology for tracking energy flow to help design processes that can meet energy efficiency targets)

Adapt PINCH technologies for use at the community level to facilitate the development of a standardized methodology for measuring energy at the community scale; to demonstrate techniques for integrating new and existing energy supply systems; and to develop energy-related guidelines for the design of sustainable neighbourhoods. $5.1M
2008-09 to 2011-12
Next Generation Technologies Sub-programs

4. Mechanical/Renewable Energy Systems

Develop prototype high-efficiency clean energy technologies, new approaches to high-efficiency distribution systems, and methods for integrating renewable energy technologies into housing and buildings. $5.6M
2008-09 to 2011-12

5. Heat Pumping Technologies

Develop new knowledge, methods and simulation tools to assist in the analysis of the integration of cooling and heating systems, mainly heat pumping technologies into houses, buildings and communities. $5.3M
2008-09 to 2011-12

6. Thermal Energy Storage

Develop seasonal storage technologies for solar energy; evaluation of the technical performance of an ambient seasonal cooling field trial using cold harbour water; and the development and testing of short- and long-term thermal storage integrated with micro-cogeneration technologies. $4.0M
2008-09 to 2011-12

7. Energy Management and Operations

Develop the knowledge, skills and tools to enable building operations practitioners to address the complex nature of commercial and institutional building operation, and to improve residential control systems to enable renewable energy technology integration and reduce peak loads. $6.0M
2008-09 to 2011-12

8. Lighting and Daylighting

R&D regarding ultra-low energy lighting systems market: advanced daylighting and facades; and solid state lighting for offices. The R&D is expected to optimize solar canopy design and prototype; advance technologies; and develop solid state lighting prototypes. $1.5M
2008-09 to 2011-12

9. Community Energy Systems

Develop methods and tools to assess the potential for the implementation of district energy (DE) systems, including those that use local low-grade energy supplies, and develop guidelines for the appropriate design of DE systems. $2.2M
2008-09 to 2011-12

10. Building Envelope Systems

Research to improve thermal performance and to ensure durability of building envelope components, including the development of specifications, innovative materials, and product assessments. $2.6M
2008-09 to 2011-12

11. Distributed Energy Resources

Research to improve the economics and efficiency of converting non-renewable energy to electricity in distributed systems focusing on cogeneration, energy storage, energy conversion and integration technologies. $1.0M
2011-12
Management and Administration; IEA projects/activities $1.8M
2008-09 to 2011-12
Total $45.9
2008-09 to 2011-12

* Includes all PERD, ecoETI, CEF ecoEII funds to projects in each sub-program plus NRCan A-base, and other NRCan cash and in-kind

1.2.6 Key Stakeholders and Beneficiaries

Stakeholders involved with the BEP include the delivery agents and the organizations with which they collaborate; and the receptors/targeted beneficiaries of the research outputs.

The primary recipients of the R&D funding and delivery agents of the R&D projects within the sub-programs are federal government departments/agencies. Federal participants include:

  • CanmetENERGY–Ottawa, CanmetENERGY–Varennes and the CanmetMATERIAL(NRCan);
  • Research Division, Canada Mortgage and Housing Corporation (CMHC);
  • Institute for Research in Construction, National Research Council (NRC);
  • The Real Property Program Branch, Public Works and Government Services Canada (PWGSC); and
  • Environment Canada (EC).

These federal departments/organizations collaborate with partners that have expertise in the research area and/or an interest in the research outputs. Partners provide funding and/or in-kind resources (e.g., cash contributions; access to facilities) and/or participate in the research projects. Technologies are typically developed, integrated, prototyped and demonstrated in partnership with a manufacturer. Research for codes and standards development is done in partnership with standards committees and regulatory bodies. Partner organizations include private and public sector research organizations, community groups, academia, other BEP sub-programs and other federal departments, provincial and municipal government departments, utilities, industry associations, building industry (private companies, consulting firms) and international agencies.

Receptors, or target beneficiaries, are expected to use the research outputs. Receptors are often also research partners. Receptors of BEP work include:

  • Private sector/industry (e.g., builders, renovators, sub-trades, design and construction engineering firms, product and systems manufacturers, developers and planners, energy technology industries, power generation industry, R&D researchers, and non-governmental organizations);
  • all levels of government and utilities (for use in policies, technology deployment programs, standards, codes, and regulations, municipal plans, community and neighbourhood level energy and development planning; research);
  • communities and community groups;
  • universities and colleges (e.g., tools for training students, results for ongoing research);
  • other sub-programs (e.g., results from next generation technologies may feed into models or testing in the three R&D integration sub-programs – NZEH, NZEB, Pinch); and
  • international agencies (e.g., International Energy Agency (IEA)).

2.0 Evaluation Approach and Methodology

2.1 Evaluation Scope

This BEP was previously evaluated in 2009 (referred to as the Buildings and Communities Energy Technology (BCET) Program) and covered the period 2003-04 to 2007-08. The approach and level of effort used in the current evaluation were based on an evaluation risk assessment. In general, the BEP programs were assessed as having a low-medium level of risk. The areas of medium risk were management practices and structures (e.g., portfolio governance structures were put into place in 2007-08), delivery complexity (e.g., PERD is an interdepartmental program, and involves multiple funding streams and delivery partners), and performance measurement (performance measurement issues were identified in the 2009 evaluation).

Given their interconnectedness; the current evaluation covered all of the sub-programs in the BEP. However, the sub-program Distributed Energy Resources received limited focus as it was included in the Evaluation of the Clean Electric Power Generation Sub-sub Activity (carried out in 2009-10), and was only situated in BEP for one year.

An Evaluation Advisory Group was established to coordinate and manage communications between program representatives (IETS, OERD, National Research Council (NRC)) and the Strategic Evaluation Division.

The evaluation includes both R&D and demonstration activitiesFootnote 38 , but is focused on R&D given that most of BEP activities involve R&D. PERD is the main funding source for the Portfolio and is focussed on R&D. C-based (i.e., time limited) programs provide funding to various Portfolios for R&D and for demonstration activities. Only a small portion of these funds were allocated to demonstration projectsFootnote 39 in this Portfolio. For example, ecoETI provided some funds for pilot-scale demonstration of clean energy technologies under the Portfolio. Some CEF demonstration funds initially falling within the renewables portfolio were reclassified under BEP subsequent to the evaluation timeframe and are therefore not within the scope of the evaluation. ecoEII demonstration projects within BEP were launched in 2012-13 and therefore not within the scope of this evaluation.

2.2 Evaluation Approach and Methods

The evaluation used multiple lines of evidence to assess relevance and performance of the Portfolio: i) document review; ii) literature review; iii) interviews; and iv) case studies. Details of each method are provided below.

In order to assess progress of longer-term outcomes, a number of the case studies involved an analysis of the evolution of research dating back to the previous cycle (2003-04 to 2007-08).

Document Review:

One hundred and twenty (120) documents were reviewed to obtain descriptive information about the programs and contextual, financial and performance information. Key documents reviewed included:

  • Treasury Board Submissions; Results-Based Management and Accountability Frameworks
  • Reports on Plans and Priorities and Departmental Performance Reports (RPPs and DPRs)
  • Portfolio and program plans; sub-program and project proposals
  • Mid-term and annual performance reports; project reports
  • Previous reviews; impact study; evaluation; and audit

Literature Review:

A review of international literature was conducted to identify best practices for R&D knowledge transfer and technology uptake.

Interviews:

In total, 112 interviews were conducted with representatives of NRCan, other federal departments and agencies and external stakeholders. Given the evaluation’s focus on uptake of knowledge and technology, the majority of interviews were conducted with external partners and receptors. The distribution of interviews among stakeholder groups was as follows:

  • NRCan representatives: OERD and IETS senior managers, managers and researchers, and other NRCan Divisions (n = 31);
  • Other federal senior managers, managers and researchers: NRC, CMHC, PWGSC, DND, EC (n = 15);
  • External stakeholders (n = 66) distributed as follows:
    • Real estate developers and builders (n = 6)
    • Industry associations (n = 9)
    • Manufacturers and distributors (n = 9)
    • Researchers at universities and provincial research institutes (n = 12)
    • Provincial representatives (n = 5)
    • Utilities (n = 4)
    • Commercial building managers and operators (n = 6)
    • Residential and building consultants, engineers, architects (n = 11)
    • Municipal representatives (n = 4)

Interviewees were selected in consultation with the Evaluation Advisory Committee to ensure representation of all stakeholder groups across all sub-program areas. Individual interviewees were identified based on suggestions from sub-program coordinators and researchers; on the evaluation team’s review of documents; and using the ‘snowball’ technique where external stakeholders were asked to provide names of other external receptors or potential receptors.

Case Studies:

A total of 12 case studies were developed to facilitate an in-depth understanding of the linkages among activities, outputs and outcomes; factors that facilitate or hinder achievement of results; best practices and lessons learned. Case studies involved a review of project level documents and interviews with federal project leads (n = 14), as well as external partners and receptors (n = 46).

Case studies were selected to ensure representation across sub-programs and research at different stages along the innovation continuum. Individual case study projects were selected in accordance with established criteria (i.e., representation across sub-programs, different types of target audiences and research projects) based on a document review and in consultation with the Evaluation Advisory Committee.

2.3 Evaluation Limitations and Mitigation Strategies

There were three key evaluation limitations: 1) The assessment of longer-term R&D outcomes within a four-year timeframe; 2) Assessment of the extent of R&D uptake; and 3) Non-representative sample of external stakeholders and receptors.

Longer-term R&D results typically have a lengthy time horizon between early-stage research and broader deployment– this process can take 15 years or longer. Using a four-year timeframe to analyze the impact of research activities limits the capture of the longer-term impacts of the research. The evaluation mitigated this challenge by:

  • Selecting case studies that have historical linkages to previous NRCan funding to facilitate analysis of the evolution of technology/knowledge transfer; and
  • Reviewing a BEP Impact Assessment StudyFootnote 40 funded by OERD and commissioned by IETS that assessed built environment research activities dating back to the 1990s.Footnote 41

It was difficult to assess the extent of knowledge and technology transfer, partly as a result of inadequate performance information (for example inconsistent reporting of outcome information) and due to the fact that the total population of end-users is not known. The evaluation mitigated this challenge by:

  • Conducting the majority of interviews with external receptors (a total of 112 external stakeholders were interviewed including those interviewed as part of case studies);
  • Conducting a detailed review of narrative performance information to enable an analysis of types of research being conducted, the extent to which projects have progressed; and identification of areas where uptake has been less than expected.

The evaluation was not able to obtain a representative sample of end-users given that the total population of end-users is not known. The wide variety of R&D outputs and end-users makes it virtually impossible for program or evaluation representatives to identify the total population. The evaluation increased the number of respondents and minimized selection bias of external receptors through the following means:

  • While many external partners and receptors were selected in consultation with the Evaluation Advisory Committee, SED also selected additional receptors upon reviewing program documents, ensuring there was representation across sub-programs and types of end-users (e.g., industry, academia, provinces).
  • The “snowball” or referral sampling technique was used to locate additional external stakeholders. This technique involved asking external stakeholders for other interviewee referrals (who were end-users or in some cases potential receptors).

3.0 Evaluation Findings

3.1 Relevance

3.1.1 Continued Need for Program

Evaluation Question Methodologies Assessment
1. Is there a continued need for the Built Environment programs and activities? Interviews
Case studies
Document and literature reviews
There is a continued need for Built Environment RD&D to fill knowledge and technology gaps that will enhance energy efficiency, reduce GHG emissions, and enhance economic competitiveness.

Summary:

There is an ongoing need for BEP programs, particularly with respect to retrofits, to respond to technology and knowledge gaps for improving energy efficiency and reducing GHG emissions in the housing and buildingFootnote 42 sectors. BEP also has the potential to contribute to reducing strain on existing utility infrastructures and to enhance economic opportunities.

The Portfolio is generally aligned with identified research and technology gaps as indicated in the literature and by internal and external interviewees. External stakeholders note that there are unmet needs related to demonstrating the cost-effectiveness and technical feasibility of advanced technologies in the built environment.

Discussion and Analysis

Interview, case study, and document evidence support the continuing need for the Built Environment R&D programming as a means of contributing to increased energy efficiency and reduced GHG emissions. In addition, BEP research focused on improved energy efficiency and peak load issues has a role to play in reducing energy consumption; which in turn reduces the strain on existing utility infrastructures.Footnote 43

There is debate in the international literature about whether RD&D investments in clean energy should be made now or in the future given the low costs of energy. While low or non-carbon emitting energy technologies, beyond niche markets, are not likely to be competitive over the next few decades, OECD argues that “learning investments” will have to be made before full competitiveness can be reached by new technologies.Footnote 44 Moreover, investments may be necessary in the near term to make future reductions in energy use and GHG emissions cheaper.Footnote 45 Longer term investment in energy efficiency technology can build innovative capacity which has the potential to provide export opportunities. Moreover, innovation should lead to cost reductions which can make it cheaper and easier to invest in energy efficiency in the future.Footnote 46

Environmental Needs

The housing and buildings sectors are major energy users and upward pressure on energy demand is expected to continue. Energy use is the principal source of greenhouse gas emissions and other air pollutants in Canada.Footnote 47 As shown in Figure 4, in 2009 the residential and commercial/institutional sectors accounted for 31% of secondary energy use 28% of the total GHG emissions from energy use by all sectors.Footnote 48 The number of households is expected to increase by 20% from 2005 to 2020, increasing future demand for energy in the residential sector.Footnote 49 Upward pressure on energy demand is also expected in the commercial/institutional sector as commercial floor space is forecasted to increase 39% between 2005 and 2020.Footnote 50

Figure 4: Secondary Energy Use in Canada by Sector

Figure 4: Secondary Energy Use in Canada by Sector

Text version

Figure 4: Secondary Energy Use in Canada by Sector

Figure 4 is a pie chart depicting secondary Energy Use in Canada by Sector in 2009. The industrial sector uses the most energy (37%), followed by transportation (30%); residential (17%); commercial (14%); and agriculture (2%). The source of this information is the Office of Energy Efficiency. NRCan. Energy Use Data Handbook.

Several documentary sources point to the opportunity and need to reduce Canada’s energy use and GHG emissions by achieving major efficiency gains in the built environment sector.Footnote 51 An IEA report notes that to achieve a significant environmental impact rapid cost reductions and substantial technical improvements are required in both existing and emerging energy technologies.Footnote 52

According to the National Energy Board, energy performance for buildings (both residential and commercial) is seen as one of the most feasible opportunities for energy savings and greenhouse gas emissions reductions. Residential use is the largest energy saving opportunity and the combined residential and commercial/institutional building opportunity accounted for 75 percent of the total opportunity for energy savings.Footnote 53 In 2006, the National Advisory Panel on Sustainable Energy Science and Technology in Canada stated that there should be a substantial increase in funding for energy science and technology in the building sector, given the potential opportunity for more efficient energy use.Footnote 54

Increased Emphasis on Retrofits

Many interviewees noted that to have the most potential to impact GHG emissions BEP activities need to increase emphasis on retrofits. During the timeframe of this evaluation, interviewees noted that the much of focus of the R&D was on new builds. However, retrofitting is being given additional focus in the 2012-13 to 2015-16 planning cycle. The need for increasing energy efficiency of existing buildings relates to the physical longevity of the built environment. As the National Roundtable on Energy and Environment (NRTEE) observed, “Canada’s building stock is a long-lasting component of its energy use infrastructure: 66 percent of the buildings that will be standing in 2050 are already built. This means that there will be a major renovation effort needed to bring these buildings up to the standards.”Footnote 55 Research is also required with respect to building operations as improved operations have the potential to increase energy efficiency.Footnote 56

Alignment of Portfolio with Knowledge and Technology Needs

There was general agreement among internal interviewees that BEP’s broad scope was aligned with key knowledge and technology needs.

On the whole, BEP activities are congruent with external stakeholder needs, as indicated by interviews, case studies and literature reviewed.Footnote 57 Key needs identified by external stakeholders and in the literature were:

  • Increase cost-effectiveness of new technologies;
  • More integrated approach to energy efficiency in the built environment to maximize reductions in energy use and realize other co-benefits;
  • Effective decision-making tools and information to help assess feasibility, energy use and costs of energy efficient technologies, systems, or methods; and
  • Improve industry capacity and competitiveness.

Document evidence indicates that a BEP’s objectives and priorities are aligned with these needs. For example, a common theme in the document and literature review was the need for an ‘integrated approach’Footnote 58 to energy efficiency in the built environment to maximize reductions in energy use and realize other co-benefits. Advanced performance is founded on the integration of systems, integrated design, and integration technologies such as simulation tools and control systems that are required to design, construct and operate them.Footnote 59 This theme was echoed by both internal and external interviewees who indicated the need for an understanding of how technologies and systems interact for both new building design and for retrofits. Building stakeholders noted they need more information on the implications of high performance buildings on operations and how to resolve issues arising from this. This theme of integration is consistent with the priorities for the BEP as expressed in their Program Plan for 2008-09 to 2011-12Footnote 60 : emphasis on integration technologies for improved efficiency; emphasis on integration of renewable and clean energy technologies; and the whole-house, -building and –community near-zero and net-zero solutions.

Some internal interviewees and many external interviewees indicated that a better understanding of the relationship between behaviour and energy efficient technology is needed and that the Portfolio could better address this need. A number of external stakeholder groups (e.g., builders, utilities) are interested in behaviour because it aligns with their needs such as marketing and development of incentive programs and green building policies. Builders consulted noted that while consumer interest in energy efficient houses and buildings has increased over the past few years, it still does not outweigh factors such as costs, style and comfort. Builders indicated that there is a link between increased energy efficiency and comfort and that they tend to use the concept of comfort to sell energy efficient products or houses.

Other internal interviewees point out that given the interdependence between behaviour and many energy efficient technologies, human factors are an important consideration for facilitating the uptake of new technology. This is supported by international research on the subject. For example, an OECD/IEA paper notes that the distinction between behavioural and technical changes may not be clear-cut as technical and behavioural elements can be closely linked. The OECD/IEA paper concludes that both behavioural and technical changes must occur together to achieve optimal energy efficiency / energy intensity gains.Footnote 61 Work addressing behaviour is being covered to a limited extent within the Portfolio (e.g., lighting, demand response strategies). Given interview and document evidence, human factors should continue to be considered within the overall context of the built environment advanced technology or system when appropriate and within the confines of existing budgets and resources.

Both internal and external interviewees noted the importance of demonstrations in building industry capacity, increasing cost-effectiveness and mitigating the risks of adoption. There was general agreement across all interviewees that there was a funding gap to support demonstrations, particularly for small demonstration projects targeted at small and medium sized enterprises (SMEs). In addition to R&D efforts, there are also known technology options that exist in energy production, conversion and end-use that could reduce emissions significantly, that would benefit from demonstration projects.Footnote 62

3.1.2 Alignment with Government Priorities and NRCan Strategic Objectives

Evaluation Question Methodologies Assessment
2. Are the Programs consistent with government priorities and NRCan strategic objectives? Interviews
Case studies
Document review
BEP R&D is consistent with government priorities relating to clean energy and economic and northern development, and NRCan’s strategic objective relating to environmental responsibility.

Summary:

BEP objectives are aligned with government priorities relating to clean energy, economic and northern development as identified in Speeches from the Throne and Budgets, and voluntary energy commitments to GHG reductions as set out in the Copenhagen Accord in 2009. BEP is also aligned with NRCan’s strategic objective relating to environmental responsibility. While BEP supports the mandates of other federal departments and agencies, there has been diminished involvement by some federal players (e.g., Environment Canada, Public Works and Government Services, and Canada Mortgage and Housing Corporation) due to shifting priorities and decreased capacities. The Department of National Defence has become an active federal partner in BEP R&D given its potential application to mobile and Northern military operations.

Discussion and Analysis

Interview, document and case study evidence indicates a general alignment between the Built Environment Portfolio and federal priorities and NRCan’s strategic outcome – Environmental Responsibility.

The Portfolio’s objectives are aligned primarily with the government’s clean energy priorities. Canada is currently maintaining a voluntary commitment to reducing GHG emissions by 17% from 2005 levels by 2020, as per the target set internationally in the Copenhagen Accord in December 2009.Footnote 63Mitigating climate change and air pollution have remained a federal priority over the evaluation period, as is evident through the Clean Air Agenda announced in 2006 and renewed in 2011. Budgets and Speeches from the Throne make reference to investments in clean energy technologies.Footnote 64

Both the Speech from the Throne and Budget 2011 highlight the deployment of clean energy technologies in Aboriginal and Northern communities.Footnote 65 BEP is aligned with Northern Development as its purpose is to improve knowledge and technologies to increase energy efficiency in the built environment, including remote communities.

The BEP objective relating to the provision of economic opportunities also aligns with government economic priorities such as those reflected in the federal Jobs and Economic Growth Act meant to “secure our country’s economic recovery, encourage growth and create jobs.”Footnote 66

Alignment with NRCan’s Strategic Objectives

NRCan’s mission and mandate encompasses the following:

  • the responsible development and use of Canada’s natural resources and the competitiveness of Canada’s natural resource products;
  • policies and programs that enhance the contribution of the natural resources sector to the economy and improve the quality of life for all Canadians; and,
  • S&T in the field of energy, especially conducting innovative science to generate ideas and transfer technologies.Footnote 67

The document review and internal interviews confirm that the BEP is aligned with NRCan’s mandate and strategic outcomes. Both focus on sustainable development of Canada’s natural resources. BEP falls under NRCan’s second strategic outcome relating to environmental responsibilityFootnote 68 and is well-aligned in promoting the development of energy efficient R&D solutions for the built environment.

Alignment with other federal departments/agencies

Interview, document and case study evidence indicates that BEP supports the mandates of other federal departments and agencies. For example, DND’s recent increased involvement in PERD is a means of addressing knowledge/technology gaps for military application in the North and for mobile applications. There has been diminished involvement of some other departments and agencies however, (i.e., Environment Canada, Public Works and Government Services, and Canadian Central Mortgage and Housing) due to shifting departmental priorities and reduced research capacities.

3.1.3 Legitimate, Appropriate and Necessary Federal Role

Evaluation Question Methodologies Assessment
3. Is there a legitimate, appropriate, and necessary role for the federal government in the Program? Interviews
Case studies
Document review
The role of the federal government and NRCan in BEP R&D is legitimate, appropriate and necessary to support regulations, codes and standards and to serve as a catalyst in developing built environment knowledge and technology.

Summary:

There is a legitimate and necessary role for the Department to conduct energy related R&D in the built environment sector as expressed in NRCan’s policy of sustainable energy use, as outlined in the Energy Efficiency Act. The federal government also has a legitimate and necessary role to develop a knowledge base needed for codes, standards and regulations. The role of the federal government in BEP R&D serves as a catalyst for R&D and is appropriate to fill important knowledge and technology gaps not being addressed. Case study and interview evidence shows the importance of BEP funding and support. External interviewees generally agree that without government support, projects would not have taken place or would have been reduced in scope. Both interview and case study evidence underscore the leadership role played by federal researchers in many R&D projects. External stakeholders value the federal role in BEP R&D and underscore that it must occur within the context of collaboration.

Discussion and Analysis

Legitimate and necessary role

There is a legitimate and necessary role for the Department to conduct energy related R&D in the built environment sector as expressed in NRCan’s policy of sustainable energy use, as outlined in the Energy Efficiency ActFootnote 69 and in NRCan’s S&T strategy.Footnote 70 The federal government also has a legitimate and necessary role to develop a knowledge base needed for codes, standards and regulations. BEP R&D provides technical input into the development of model national energy codes for housing and buildings, and the Canadian Standards Association which contribute to the harmonization of codes and the reduction of market barriers. There was general agreement that the NRCan role as lead on energy technology is appropriate given its energy efficiency mandate.

Many provinces/territories have implemented various regulatory- or voluntary based energy efficiency initiatives in houses and buildings. For example, British Columbia, Alberta, Manitoba, New Brunswick, and Quebec have energy efficiency incentive programs in place.Footnote 71 BEP has a fundamental role to play in supporting science and research underpinning regulatory systemsFootnote 72 and can inform energy efficiency policy and programs. R&D is required to inform regulation and public policy in important areas such as environmental protection which is aligned with energy issues.

Appropriate Role

Internal and external interviewees agreed that the coordinated actions of both federal and provincial stakeholders are needed to enhance energy efficiency and reduce GHG emissions.

Community representatives also highlighted the importance of coordinated action of federal and provincial /territorial governments. A recent report prepared by NRCan on governance issues in community energy planning in Canada underlines the value of NRCan’s role in community energy and the built environment. The report discusses conditions for success if community energy solutions are to become a major element of Canada’s approach to energy efficiency and GHG reductions. It goes on to state that the key to success from the national perspective is coordinated action of federal and provincial governments.Footnote 73 According to the report, a critical element of such coordinated action would be establishing a national role for NRCan in research and development, coordination, technical support and communication of best practice.Footnote 74

All lines of evidence indicate that the federal government, in collaboration with others, plays the important and appropriate roles of: 1) facilitator and catalyst in the innovation system; and 2) direct R&D performer and funder. External interviewees generally agree that the federal role as a direct R&D performer and funder is appropriate given industry’s lack of capacity and the longstanding federal expertise in built environment R&D. The Intergovernmental Panel on Climate Change (IPCC) estimates that a third of advanced technologies (that have potential to substantially reduce emissions) have discounted paybacks of 10 years or less, and noted that government needs to provide support and investment for more cost-effective technologies and systems.Footnote 75

Interview and case study evidence support that, through BEP, NRCan and other government departments play a leadership / catalyst role in a range of clean energy initiatives, providing core funding, expertise, facilities and credibility that attract participation by other Canadian and international stakeholders. Longstanding federal expertise in energy efficient houses and buildings and ability to bring key stakeholders together were cited by both internal and external interviewees as key enablers for innovation. BEP provides a number of federal leadership examples (e.g., Drake Landing Solar project; zoned comfort systems; solar driven liquid desiccant air cooling).

Interview and case study evidence support that BEP provides continuity supported by long-standing expertise developed in government labs. The federal government role is perceived by interviewees as most appropriate for longer-term or high risk research. However, some suggest the federal government role as more appropriate for near market research.

Interviewees agreed that the federal role in supporting demonstration projects in the built environment was appropriate. The literature review indicated that there is a greater need for funding demonstrations in the energy sector as compared to other sectors due to high investment needs, low capital turnover, and competition from existing, mature technologies. New technologies may have better operating performance, but high capital costs are a significant barrier to demonstration and therefore the federal role is considered to be warranted.Footnote 76

External stakeholders note that there is little R&D funding available from other sources and that there has been a decline in funding, particularly since the economic downturn in 2008, making federal support for R&D even more important because it facilitates the continued enhancement of private sector and external stakeholder capacity.

Given intellectual property issues and a reluctance to share R&D among competitors, industrial partners emphasize that having government as a neutral knowledge disseminator and network facilitator is critical. Moreover, many external stakeholders note that federal support provides enhanced credibility and visibility to the R&D which helps stimulate public / consumer interest in energy efficiency.

Evidence of Duplication

Case study and interview evidence indicate that there is minimal duplication between BEP and other federal or provincial, industry or academic research programs. Across federal government departments involved in PERD, the evaluation found no clear evidence of duplication. While both NRCan and NRC are involved in high performance buildings research, internal interviewees note that strong planning and communication links between the two organizations ensure that overlap is avoided.

External interviewees note that where there are other federal R&D programs pertaining to energy efficiency and the built environment (e.g., Sustainable Development Technology Canada; NSERC), they tend to be complementary and do not have the same scope or objectives as BEP. External interviewees note that NRCan can play an important role in building the capacity of SMEs through smaller projects in comparison to SDTC which external stakeholders consider is more appropriate for supporting larger projects. Academic interviewees noted that NSERC is a strong supporter of university research and tends to provide funding for programs in earlier stages of research.

Academic interviewees indicated that NRCan and other PERD government players are particularly valuable for facilitating the transition from earlier stage research to an applied field trial or demonstration. Other industry, provincial and municipal stakeholders note that research in energy efficient houses and buildings is limited and has decreased over the past few years, with industry lacking the capacity to conduct research.

Additionally, parallel activities with the IEA help to avoid duplication at the international level. Internal documents indicate that good linkages exist among research performers in the Built Environment Portfolio and the international research.Footnote 77 BEP supports five IEA Annexes and plays a strong role in a number of Implementing Agreements.Footnote 78

3.2 Performance

3.2.1 Achievement of Expected Outcomes (Effectiveness)

Evaluation Question Methodologies Assessment
4. To what extent have intended outcomes been achieved as a result of the BEP programs? Interviews
Case Studies
Document Review
Literature Review
BEP R&D contributes to both immediate and intermediate outcomes and is making progress towards longer-term outcomes, particularly with respect to energy efficient housing and some advanced technologies

Summary:

BEP R&D contributes to broader deployment of energy efficient technologies and practices through its technical input into codes, standards, policies and programming, particularly OEE programs. There have been good examples of uptake on the housing side and for several next generation technology sub-programs. Projects targeting buildings and communities have made progress in developing capacity among target groups and in piloting advanced technologies and tools. While the Portfolio as a whole is making good progress towards its immediate and intermediate outcomes, the achievement of longer-term outcomes such as broad deployment are constrained by low energy prices, particularly natural gas prices, and the absence of other enabling conditions (e.g., supportive policies and programs).

Discussion and Analysis

3.2.1.1 Increased Capacity

Evidence indicates that external partners have enhanced their awareness, knowledge and skills as a result of involvement in BEP projects. While there has been broader capacity enhancement among some end-user groups, limited performance information (e.g., inconsistent reporting of outcomes) and evaluation limitationsFootnote 79 makes it difficult to assess the extent of achievement. Evidence indicates that while dissemination is occurring the lack of a comprehensive strategy for dissemination (and inadequate resources) likely influences capacity enhancements to broader groups of end-users. Moreover, interview evidence suggests there are some inconsistencies in the extent of engagement of external partners. For example, some external partners noted that they were not adequately debriefed at the completion of the research project. Other external partners noted that research results were not appropriately tailored to various audiences. Improved communication and dissemination to industry groups and associations (e.g., particularly in relation to buildings and communities) could be further enhanced.Footnote 80

BEP Outputs and Dissemination Activities

In addition to interview and case study evidence, proxy indicators, such as number and type of outputs and dissemination activities, have been used to assess achievement of this increased capacity. BEP has produced outputs in a variety of forms. According to BEP internal documentation, 459 publications/reports have been disseminated through the following means during the evaluation timeframe:

  • 72 refereed journal articles;
  • 80 non-rfereed journal articles;
  • 157 conference proceedings/ papers;
  • 117 internal technical reports; and
  • 33 client reports.Footnote 81

Approximately 70% of these outputs were disseminated to a range of audiences (i.e., through published papers or conference proceedings), and to the extent that this information is accessed and used, it would potentially contribute to capacity enhancement. In addition, 277 domestic partners and 107 international partners, and 84 graduate and post-graduate students were involved in BEP projects over the evaluation period.

A review of CanmetENERGY’s websiteFootnote 82 provides general information on areas of current research for buildings, housing and communities as well as access to BEP information materials, including publications, case studies, factsheets, training materials, reports and analyses. However, the website does not contain a comprehensive list or links to all BEP publications. Challenges relating to dissemination via the Internet are discussed later in the report.

The evaluation found that research outputs did not consistently reach their target audiences due the lack of a comprehensive dissemination strategy. External and internal interviewees noted that, overall, there were insufficient resources allocated to dissemination and that this was a key factor hindering ability to reach broader end-user groups. External interviewees also pointed to a need for increased capacity within CanmetENERGY to develop less technical communications products that are tailored to different audience needs (for example building operators, utilities, provinces, industry associations, and general public).

A number of interviewees suggested strengthening engagement with industry associations to aid in dissemination efforts to facilitate broader dissemination. External interviewees did note however that CanmetENERGY’s housing unit in particular was connecting with appropriate associations, such as the Canadian Home Builders Association and the NetZero Energy Coalition. A document review revealed other connections between BEP researchers and various Associations. For example, BEP researchers are involved in providing advice, participating in review processesFootnote 83 or in task forces/committeesFootnote 84 . On the buildings side, some external stakeholders indicated that Real Property Association of Canada (REALpac) was one of the primary sites they used to obtain information on high performance buildings and recommended that NRCan/ CanmetENERGY strengthen linkages with this organization.

Some building stakeholders indicated limited awareness of BEP building related outputs. Community sector stakeholders consulted reported good awareness of NRCan’s involvement and expertise in community relevant research. Moreover, the document review and interviews revealed linkages with appropriate associations (e.g., Canadian Federation of Municipalities (CFM), Quality Urban Energy Systems of Tomorrow (QUEST) and the Canadian District Energy Association (CDEA). However, some community stakeholders indicated that communications between BEP researchers and pertinent associations could be enhanced.

A review of some key industry association websites (Building Owners and Managers Association (BOMA), Real Property Association of Canada (REALpac), Canadian Home Builders’ Association (CHBA), Canada Green Building Council (CaGBC), Federation of Canadian Municipalities (FCM), American Society of Heating, Refrigeration and Air-Conditioning Engineers (ASHRAE) showed that with the exception of CHBA, REALpac, and CaGBC there were no apparent links, inclusions or references to BEP research. The CHBA website contained references to some NRCan R&D information (e.g., slide presentations). The CaGBC research provided a link to NRC’s Post Occupancy Evaluation, which was partially funded through PERD. Only one outdated reference to BEP was located on the REALpac website - a 2005 publication with reference to CanmetENERGY research outputs (e.g., EE4 software).Footnote 85 However, some of these sites (for example ASHRAE, Illuminating Engineering Society North America (IESNA)Footnote 86 do contain references to work partially supported by PERD. These references are in citation lists within individual guideline documents rather than direct links on websites.

Knowledge/Technology Advancements

Case study, interview and document evidence suggests that projects are progressing along the innovation continuum, producing key knowledge gains or technology enhancements. However, given the inconsistencies in performance reporting, it is difficult to assess the extent to which projects have produced significant knowledge or technology advancements. A review of PERD/ecoETI progress reports indicate progress for many projects as indicated below:

An SED analysis of PERD/ecoETI progress reports identified the following outputs: 60 guidance documents (e.g., technology assessments, plans and specifications, design guidelines, best practice guides); 28 enhanced models, simulations or software (e.g., Spatial Community Energy Carbon and Cost, Regional Energy Analysis Model, Hot3000) for use by researchers and end-users; 27 data files; 15 large-scale prototypes; 53 field trials; 7 enhanced processes or methodologies; and 6 designs or construction specifications.

  • New or enhanced processes / methods cited in 7 instances in the progress reports (e.g., process modifications to cogeneration unit so that it operates at an improved performance level; new testing analysis methods of fuel cell and battery management systems; enhancement of community level demand methodology)
  • New or enhanced modeling capacity cited in 28 instances. In many cases predictive models were validated using research results. Many of these validated models were used by external and internal research for further R&D (e.g., validated models of the CCHT house, cogeneration unit, thermally driven cooling systems; development and validation of several elements of TRNSYS model ( some promising near and net zero housing options; incorporating target efficiency levels for code development support; solar-driven liquid desiccant air conditioning system; water storage components; solar seasonal energy storage model; heat pump/ice slurry experimental work);
  • Decision-making tools such as HOT3000 and CanQuest were beta tested during the evaluation time period (although delays in development of these tools were noted). Community based tools such as Spatial Community Energy Carbon and Cost (SCEC3) and Regional Energy Analysis Model (REAM) were piloted in British Columbia.
  • Improvements in technology performance cited in many projects: Performance enhancements (e.g., energy savings) cited in a number of projects (e.g., solar canopy system; microblind; phase one ejector prototype; controllability of LED lighting; and Photovoltaic shading device prototypes and control strategies). Testing and adaptation/development of existing technologies to Canadian context common in Next Generation Mechanicals and Renewables and Distributed Energy Resources sub-programs (e.g., testing of cold climate mini split heat pumps, combination space/ water heating system based on a tankless water heater, hybrid radiant/forced convection heating system prototype for buildings).

Strengthened Capacity of External Partners

Interview, case study and documentary evidence indicates that the capacity of external partners was enhanced through involvement in BEP projects. The majority of external interviewees reported that their capacity was enhanced in a variety of ways:

  • New or strengthened collaborations and partnerships;
  • Enhanced reputation of company;
  • Increased awareness of benefits of validated models, accurate measurement of energy use;
  • Increased knowledge of viability of energy efficient methods and technologies; integrating technologies; product installation; and the impact of technology on behavior;
  • Increased ability to conduct IDP (integrated design processes); and
  • Better prepared for new code updates.

Other external researchers noted that BEP provided them with access to data which improved the scope and quality of their research. Provincial partners noted that the expertise and information from BEP was useful for decision-making and policy development. Some external partners noted that BEP involvement also increased their ability to communicate effectively about energy efficiency practices. Academic stakeholders reported increased capacity in terms of advanced expertise in their field or a new field, particularly on the more applied side. They also indicated that access to data was important for building on their own research and validating models. Some university partners noted that BEP research also enriched their teaching activities.

Interview and case study evidence show that in a minority of cases external partners reported no increased capacity. In those cases reasons cited were: insufficient debriefing to external partners by experts; insufficient resources for dissemination; reports too technical; and research too long-term to be useful.

In addition to BEP project partners, there was some evidence from external interviewees to suggest that capacity enhancement extended beyond those parties or organizations that were directly involved in BEP research projects. Many external partners (e.g., academics, builders) noted that they were involved in additional dissemination activities to share BEP knowledge. Industry association representatives for both the housing and renewable sectors reported that work with Canmet ENERGY has led to increased capacity among association members.

3.2.1.2 BEP Influence on Policy Makers, Standards Associations, and Government

OEE regulatory and voluntary labeling programs

BEP R&D contributes to OEE energy efficiency policy and programming in the form of expertise, technologies, and models & tools for simulations, and technology assessments. This is facilitated through strong linkages and communication between CanmetENERGY and the OEE- Housing and Equipment Divisions. Knowledge derived from the R&D has been important in consumer energy product labelling and rating systems (e.g., windows).Footnote 87 BEP work is viewed as critical because it provides a strong technical base for program standards and assessment tools.

Interview evidence suggests that BEP’s contribution to OEE programs, while important, could be enhanced by ensuring that BEP project delays are minimized so that BEP work progresses sufficiently during the PERD cycle or that the PERD cycle be lengthened to provide adequate time for work to progress sufficiently for OEE information needs.

NRCan’s Impact Assessment StudyFootnote 88 notes that NRCan research and development has supported the normative structure for buildings for 20 to 25 years. Program documentation dating back to 1995 notes that PERD has contributed in the elaboration of several building codes, standards and guidelines in the areas of air leakage control, ventilation, durability, indoor air quality, commercial earth energy systems, visual performance criteria into lighting design and energy efficiency.

OEE interviewees consider that BEP technical advice for technology assessment and for standards is needed to support the Energy Efficiency Regulations. BEP provides information the Equipment Division can use to develop test procedures that can capture energy performance of a product, develop minimum energy performance standards (MEPS), and to determine product compliance with a MEPS.

According to interviewees, BEP research and expertise also contribute to OEE’s voluntary labeling programs such as: The R-2000 Home, ENERGY STAR, and EnerGuide. BEP research and expertise contributed to the development and the regular updates of these programs’ standards and assessment tools. The R-2000 standard continues to evolve based on new BEP research and testing. As recent examples, documentation indicates that BEP work under NZEH provided a framework to guide revisions to the R-2000 standard. Also, interviewees indicated that BEP work on advanced wall systems has been important for setting new targets and guidelines for wall performance in the 2012 R-2000 standard update.

In addition, the R-2000 home certification process uses HOT2000 software, developed by CanmetENERGY, to check homes compliance with the energy target set by the R-2000 standard. The HOT2000 tool is itself continuously updated with BEP R&D results. As an example, documentation indicates that the Archetypes Library for Residential / Commercial-Institutional Energy Models developed under BEP was used in the development of models for estimating energy use in houses and in commercial/ institutional buildings, including HOT2000.

National Building Codes

Evidence shows that BEP contributed to the development of the National Building Code of Canada, through its technical input into the 2011 National Energy Code of Canada for Buildings (NECB). BEP researchers also sit on various subcommittees as technical experts such as on the building and fire codes committee, the joint task force for energy efficiency in housing and buildings, Task Group on Lighting and Electrical Power, and the compliance/performance committee. Recent updates to the NECB are considered a milestone as it is being specifically designed for energy savings and is based on a whole building system approach.

Documentation indicates that the building energy models used as the reference buildings underlying the 2011 National Energy Code of Canada for Buildings, were developed based on the commercial/institutional archetypes from the Archetypes Library for Residential / Commercial-Institutional Energy Models developed by BEP R&D. All these models are expected to be used in the update of the National Energy Code for Buildings in 2016.

The NECB plays an important role in harmonization. Having engaged key actors, the development process of the new National Energy Code of Canada for Buildings (NECB) 2011 led to better adoption prospects by provinces/territories than its 1997 predecessor the Model National Energy Code for Buildings.

Contribution to Canadian Standards Association (CSA)

BEP contributed to the development of five CSA standards during the timeframe of the evaluation. A field trial conducted by CanmetENERGY in cooperation with OEE and U.S. partners was used by the standards committee to revise the CSA’s P.3 standard pertaining to the efficiency of hot water heaters. The field trial assessed hot water usage patterns in Canadian homes and was used to develop a new test procedure which allows more efficient technologies to show actual performance advantages compared to conventional water heaters.Footnote 89 BEP work under Next Generation Mechanical and Renewable Energy Systems Sub-program developed a method for determining the performance of combined space and water heating systems (combos). This work fed into the CSA P.9 standard development. In the third instance, BEP R&D contributed to a revision of the CSA P.4.1 standard – the Gas Fireplace Performance Standard.

Under the Heat Pump Sub-program, work has contributed to the development of CSA C448, Series 13 (Design and Installation of Earth Energy Systems) and CSA C748 (Performance of direct –expansion (DX) ground-source heat pumps.)

Contribution to International Standards

BEP researchers participate in numerous international organizations, including the International Standards Organization (ISO). Documentation indicates that supported by knowledge from BEP-NZEB sub-program, PWGSC provided technical expertise and input into the development of the ISO/TC 59/SC17 - Sustainability in Building Construction. BEP researchers also sit on the American Society of Heating, Refrigerating and Air Conditioning Engineers (ASHRAE) committees for the purposes of standards harmonization with the US. The Heat Pump sub-program, within BEP, also contributed to the development of two ASHRAE standards:

  • ASHRAE Standard 41.9 – Standard Methods for Volatile Refrigerant Mass Flow Measurement Using Calorimeters; and
  • ASHRAE Standard 41.10 –Standard Methods for Refrigerant Mass Flow Measurement Using Flowmeters.

Influence on Other Federal Departments

There is a limited amount of information as to BEP influence on other federal government organizations in terms of policy and programming. There is some evidence that PWGSC – a key potential receptor of building related BEP R&D – has used BEP information. For example, supported by BEP R&D, PWGSC had input in the development of clauses (regarding government buildings) for national and international Standards Associations, including: CSA, ISO, CIB, and IEA. Moreover, PWGSC’s participation in these organizations contributed to the review of its Sustainable Building Policy.

However, since PWGSC research capacity was reduced in 2010, there is an increased risk that their receptor capacity has diminished. Similarly, the diminished capacity and shifting priorities of other federal departments and agencies (CMHC, EC) may weaken their receptor capacity to disseminate and deploy BEP information for policy and program development.

3.2.1.3 Increased Use and Deployment of BEP Knowledge and Technologies

Progress has been made in the use and deployment of BEP knowledge and technologies. As was mentioned previously, BEP R&D has provided technical input into codes, standards and energy efficiency programs (e.g., R-2000) which have had reasonably broad uptake. Between 2008 and 2012, a total of 2,444 R-2000 houses were certified in Canada.Footnote 90

Longer-term achievements in the Built Environment must be viewed within the current context of low energy prices, particularly natural gas prices. Assessment of the extent of uptake should also include consideration of where research is situated along the innovation continuum. Almost half of PERD and ecoETI research (46 percent) involved the testing of large-scale prototypes and field trials. Approximately 53 field trials have been conducted from 2008-09 to 2011-12 indicating that a considerable portion of the research done is in the development stage.

Evaluation evidence suggests that there is good uptake of BEP knowledge among external project partners. The majority of external partners interviewed indicated that they were using BEP knowledge and project involvement in a variety of ways. Most developers/builders, particularly on the housing side, reported increased use of energy efficient building design and construction practices as a result of their involvement in BEP. Industry stakeholders generally indicated that they used BEP project involvement to promote and market their companies. Academic stakeholders consulted reported that BEP research facilitated their decisions as to future research directions. Many also highlighted that the BEP project gave them access to databases and monitoring files which were critical sources of information for their research. Manufacturers indicated development of more energy efficient products and equipment and improved installation of products.

Some industry associations noted that BEP knowledge was used in their planning and priority-setting processes. A few industry association interviewees pointed to a need for BEP outputs to be brought closer to commercialization in order to increase their uptake. This would involve activities such as conducting more demonstration projects, pilots and economic studies once technological soundness has been established. A few other industry association interviewees also identified a lack of capacity among their members to understand and implement BEP outputs.

Case study evidence shows evidence of good uptake of research results in 8 of the 12 case studies. In most cases where there was modest uptake it was because the project or research was at an early stage and as such insufficient time had elapsed for realization of more substantial uptake. Table 5 below summarizes case study evidence of uptake and influencing factors. For more information on the Case Studies see Annex A.

Table 5: Case Studies: Use of BEP Knowledge

Extent of Use Rating Evidence of Use of Research Some Key Factors Influencing Use/ Potential for Use
Case Study 1: Post-Occupancy Evaluation (POE) of Green Buildings led by NRC
Good: External partners (target groups) have used results External partners reported that they have used the research results to assess and make decisions about their green building policies and practices
  • Engaging the ‘right’ partners from project planning to completion. Incentives for partners to participate (access to building data and analysis)
  • NRC research reputation
  • Energy use data inadequacies (mitigated with use of secondary data)
Case Study 2: Improving the Energy Efficiency of Existing Government Buildings (development of an Office Buildings Energy Prediction Model (OBEPM)) led by PWGSC
Modest: Not yet used by target groups Model not yet launched / still in development
  • The model was field tested in nine buildings and showed potential
  • Used as a training tool for engineering students at McMaster University
  • Refinement of model continuing
  • Regular and strong communications between partners and PWGSC
  • Received limited/inconsistent performance data on building energy consumption (additional analysis conducted to mitigate)
Case Study 3: A System-Wide Methodology For Optimising Renewable Energy Solutions (SMORES) led by NRCan
Modest: Not used by external partners or target groups, some follow-up research
  • SMORES work on change management has opened up new areas of research – e.g., The Transition to Low Carbon Communities follow-up on the urban development approval process
  • Good mix of external partners
  • Inadequate energy land-use data for purposes of study /Delays due to contracting, additional administrative tasks
  • Research considered too long-term and technical for immediate uptake
  • While some dissemination occurred, hampered by limited resources & publication protocols
Case Study 4: Centrally Forced Air System Development led by NRCan
Good: System commercialized
  • A Toronto residential building company (Arista Homes) is building a 117 home sub-division with an upgrade option for zoned comfort systems
  • Increased sales of zoned comfort system industry partner. A Canadian manufacturer developed a competitive product
  • Influenced a utility’s thermostat intervention plans
  • During evaluation period, no evidence of influence on provincial policies or programs, but there were follow-on research projects with province
  • Sustained research effort and optimal use of different types of funding to move technology along continuum
  • Engaging the ‘right’ partners from planning onwards
  • NRCan leadership/expertise
  • Research addressed technical, marketing, and consumers interests (e.g., comfort)
  • Collaboration across sub-programs was key
  • Proactive approach: made presentations to key groups to ease potential barriers
  • Dissemination efforts beyond Ontario limited by resource constraints
Case Study 5: Drake Landing Solar Demonstration Project Follow-up Activities led by NRCan
Good: Project partners have used knowledge
  • Analysis of the performance data led to technological enhancements to improve system performance
  • Project partners are using results to improve products (solar collectors) and energy efficient practices and for marketing
  • Municipality has new design protocols / solar installations on facilities modelled after Drake Landing
  • Monitoring data and design model used by the research community
  • Plans for larger solar demonstration in Alberta not realized. However, feasibility of demonstration in Whitehorse underway
  • Proactive approach: In depth front-end work (e.g. review of best practices in Europe, testing equipment before installation)
  • NRCan leadership/expertise
  • Right partners (with appropriate expertise and motivation) actively involved from planning onwards
  • Formation of a non-profit company to operate and maintain Drake Landing
  • Implementation of a larger scale demonstration project limited by low natural gas prices
Case Study 6: Diagnostic Agent for Building Operation (DABO) led by NRCan
Good: Product commercialised
  • Building operators used DABO to make decisions about retrofitting and ongoing commissioning
  • DABO has been integrated into NRCan’s Low Carbon Initiative and is installed at all NRCan forestry centres.
  • DABO licensed with 30 licenses sold
  • DABO has been installed in China, Belgium, France, Australia and Canada
  • Sustained research efforts in this area
  • NRCan expertise/leadership
  • Appropriate mix of partners
  • Training and support provided by NRCan for demonstration projects
  • Demonstrated energy savings/ stakeholders positive about savings derived from DABO
  • Indications that continuing work needs to be done to enhance “user-friendliness”
Case Study 7: Optimization of Domestic Hot Water (DHW) Systems led by NRCan
Good: Uptake for most research outputs Research results incorporated into CSA standards
  • The results from the updated domestic hot water use profile are being incorporated into CSA P.3 Standard and CSA P.4.1 (Gas Fireplace Performance Standard)
  • Good collaboration between OEE and CanmetENERGY
  • Good engagement of utility partners
  • Good infrastructure to test DHW systems (i.e., CCHT). Prototype combined fireplace-condensing water heating delayed due to challenges finding industry partner
Case Study 8: Advanced Thermal Heat Pumping (HPT) based on Ejector Systems led by NRCan
Good: Research results used by project partners / other stakeholders to advance heat pump /ejector research
  • Results used to improve technology design and validate models
  • Application for a patent granted
  • Canadian manufacturer is field testing prototype ejector cooling system
  • Generated increased interest of stakeholders leading to new partnerships / further research projects
  • Sustained research efforts
  • NRCan leadership/expertise
  • Multi-disciplinary approach to research
  • Research is earlier stage, technology demonstration and transfer are needed before broader deployment possible
Case Study 9: Local Energy Efficiency Partnerships (LEEP) and Technology Adoption Pilot (TAP) led by NRCan
Good: Project partners used research outputs
  • CanmetENERGY has applied approach used in LEEP to engage stakeholders, particularly builders in other initiatives
  • Partner builders used technology assessments to identify which technology to use in demonstration homes
  • Examples of broader use of trialed technologies by builders in their subsequent work
  • Manufacturers training trades-people /building officials
  • The builder-led, collective approach, with third party technical resources facilitated technology transfer and capacity building
  • Good collaboration between BEP sub-programs
  • Significant level of participation of key stakeholders to maximize learning
  • Limited funds for dissemination/ publicity
  • Use of local suppliers/ manufacturers to engage builders
Case Study 10: Northern Sustainable Housing led by CMHC
Good: Project results used by project partners and other key stakeholders
  • Project knowledge has influenced technical and design decisions partners in subsequent northern housing projects
  • Facilitated NWT Housing Corporation’s decision to set higher energy efficiency goals for new buildings
  • Selected one of the trialed technologies (structural integrated panels) for 141 housing units
  • Tr’ondek Hwech’in used knowledge and designs as basis for further 175 energy efficient/ super insulated homes
  • Meaningful participation of all partners including local partners from planning onwards
  • The challenges of construction in the North
Case Study 11: Solid State Lighting (SSL) in the Office of the Future led by NRC
Modest : R&D shows good potential, but early stage.
  • Findings have contributed to the work of 4 international standard committees
  • The project led to follow-on ecoEII project
  • Used by a participating utility to support new incentive programs
  • SSL technology at early stage of development
  • NRC reputation/ expertise in SSL
  • Consortium structure and involvement of key stakeholders from planning onwards
Case Study 12: Advanced Durable Wall Systems: Identification of Wall Frame Assemblies to Meet Net-Zero Energy Housing Targets led by NRCan
Modest: Some use by external partners, results promising, delays in completion of outputs
  • Project results contributing to follow-on projects (e.g., Canadian Construction Material Centre test guide for Vacuum Insulated Panels (VIPs))
  • Industry partner secured IRAP funding to advance VIP technology
  • Research findings used by partner for Architectural Courses (e.g., UBC)
  • VIP is an emerging technology
  • Integrated structure of the Building Envelope Sub-program supported effective collaboration among NRCan, NRC and CMHC (i.e., individual projects were interdependent)
  • Loss of CMHC research capacity considered to be key limiting factor regarding amount and type of future research in this area
  • Completion of best practice guide and construction modules delayed to 2013 (delays in obtaining monitoring results)
  • Series of workshops on results across the country

The next few subsections highlight progress of the various sub-programs in the use and deployment of BEP R&D.

Near and Net-Zero Energy Housing (Use and Implementation)

The sub-program demonstrated very good progress in terms of increased use and deployment, and has had strong influence, particularly on early adopters. Many of the external partners (builders, developers) interviewed reported that involvement in BEP R&D projects had contributed to designing and building more energy efficient houses. For example, builders involved in the Equilibrium Housing InitiativeFootnote 91 have begun to offer near or net-zero housing products based on the experience they gained in the initiative.Footnote 92

According to interview, document and case study evidence, the builder-driven technology assessment process developed under the Local Energy Efficiency Partnerships (LEEP) project helped to speed the market adoption of energy efficient innovations (e.g., Zoning Comfort System) produced by program research. LEEP contributed to the development of 40 demonstration homes (Technology Adoption Pilot - TAP) that field-trialed energy efficient technologies selected by builders.

With respect to work conducted on the design and development of sustainable housing in the North, project knowledge was used for technical and design decisions in subsequent housing projects. Additionally the project findings were used by the Northwest Territories legislature to set higher energy efficiency goals for new buildings. The Northern Sustainable Homes modelling work led to Nunavut selecting structural insulated panels for more than 100 housing units.

Interviewees attributed the success of this sub-program in part to good linkages between CanmetENERGY and OEE Housing and Equipment Divisions. In addition, there is a well-established relationship with the Canadian Home Builders Association and the Net Zero Housing Coalition providing a good mechanism for R&D knowledge and technology development to be disseminated to a broader audience. The Canadian Centre for Housing Technologies (CCHT) provides the infrastructure for testing energy efficient housing technologies and facilitates strong linkages with NRC and the CMHC.

Near and Net-Zero Energy Buildings (Use and Implementation)

In addition to contributions of this sub-program to the National Energy Code of Canada for Buildings (NECB), progress was made with respect to the integrated design project for the NRCan Materials Technology Lab (MTL) in Hamilton, Ontario. NRCan contributed to the design of the LEED platinum MTL through provision of expertise on the integrated design process (IDP). The external partner subsequently built a second LEED platinum building using the IDP and knowledge gained from the initial MTL project.

The sub-program continues to develop a number of tools to support design of energy efficient buildings such as CanQuest and the Office Buildings Energy Predictions Model. CanQuest is based on US software tool eQuest and is being adapted for Canada. It aims to support building energy design, analysis and energy code compliance. It will also enable detailed hourly energy modeling of various technologies such as photovoltaic systems and thermal storage.Footnote 93 The Office Buildings Energy Predictions ModelFootnote 94 , a screening methodology, was applied to a sample of nine government buildings for evaluation. Case study evidence indicated that while the tool shows potential, development is continuing in the next BEP cycle.

Effectiveness of this sub-program was somewhat hindered by challenges in securing field tests sites and maintaining the engagement of external partners in some projects. For example, in one project involving the design and monitoring strategies to maximize energy efficiency in multi-unit residential buildings the developer and municipal government could not continue their involvement due to financial and resource constraints.

Communities (PINCH, Next Generation Community Energy Systems)

With respect to R&D directed at the community level, there has been some progress in the development and use of community energy characterization, mapping and costing tools in pilot projects, particularly in British Columbia and to some extent in Ontario. CanmetENERGY has made contributions to community planning by developing a more consistent method for characterizing energy and emissions in the building stock in communities.Footnote 95 R&D directed at this level is greatly facilitated by supportive provincial policies and programming, such as the introduction of the 2008 Local Government (Green Communities) Statutes Amendment Act (Bill 27, 2008) in British Columbia which introduced changes that require local governments to include GHG reduction targets, policies and plans for action in their Official Community Plans by 2010, and to set energy requirements for new developments.Footnote 96

The Spatial Community Energy, Cost and Carbon Characterization (SCEC3) model was developed from 2008 to 2012 by NRCan’s CanmetENERGY and Vive le Monde mapping in collaboration with the City of Prince George. The model enables evaluation of the energy, greenhouse gas and cost implications of specific actions related to energy use and supply in residential buildings and was intended to provide decision support for community level energy and emission reduction planning initiatives. The information derived from the model was integrated into the City of Prince George Official Community Plan.

The Tract and Neighbourhood Data Modelling Project (TaNDM) was a collaborative project between NRCan and the Government of British Columbia to develop processes and methods to obtain building attribute data. The TaNDM research project was initiated in 2011 to respond to the request of B.C.’s Community Energy and Emissions Inventory (CEEI) users to improve the aggregation of data by reporting at a finer scale. TaNDM worked to improve access and quality of buildings data for CEEI using a collaborative process and testing new methods for collecting and analyzing data. The TaNDM project resulted in the development of electricity and natural gas inventories for the building stock for two BC communities. Updates to BC Hydro’s Policy Impact Estimator are planned arising from the work done on (SCEC3) and TaNDM.Footnote 97

BC Hydro used the Regional Energy Analysis Model (REAM), a long-term energy planning tool, to evaluate the entire Haida Gwaii grid. Program documentation indicates that the District Energy Economic Model (DEEM) was used to evaluate four district energy systems in Canada (i.e. Windsor, Markham, Revelstoke, and Saint John).

The EQuilibrium™ Communities Initiative, a collaborative effort between NRCan and CMHC, assisted five projects in integrating their community planning and improving the energy efficiency of their buildings and energy systems.Footnote 98 In addition to funding, NRCan contributed to this initiative through assistance with the development of integrated community energy system concepts.Footnote 99 According to project documentationFootnote 100 , the initiative led to increased developer interest in neighbourhoods rather than single buildings. While external project partners noted numerous benefits of this initiative (e.g., enhanced capacity to design and develop sustainable communities), significant delays in NRCan’s administration of the contribution agreements created challenges for partners. The delays resulted in reduced marketing and decreased visibility, which in turn impeded ability to leverage other funds for these projects.

While the Community PINCH and the Next Generation Community Energy Systems sub-programs made progress in a number of areas, their effectiveness was limited in some cases by insufficient resources for dissemination (e.g., as indicated in the case study the “System-wide Methodology for Optimising Renewable Energy Solutions” (SMORES) project and the Compendium Manual for Community Energy Systems).

Next Generation Technologies

There has been notable progress along the innovation continuum for a number of technology areas such as the zoning comfort system, ejectors/heat pumps; solar desiccant cooling system; controllability of microblind and LED lighting; cold climate mini-split heat pumps; solar canopy system; condensing rooftop systems; and microgeneration.Footnote 101 External project partners in these sub-programs reported use of BEP for spin-off research projects; reviewing feasibility/ viability studies of new/enhanced technologies to decide whether to pursue further research; and for product improvement.

A considerable portion of the research work in the New Generation Technology sub-programs (e.g., in the Next Generation Mechanicals and Renewables and Distributed Energy Sub-programs) involves the development and enhancement of existing technologies for the Canadian context. In some instances, the adaptation of these technologies can involve innovative configurations. For example, thermal photovoltaic (TPV) and thermoelectric generator (TEG) power converters were successfully integrated with both existing and new boilers/furnaces.

The solar driven liquid desiccant project was initiated by NRCan subsequent to preliminary studies that identified liquid desiccant as a good approach for a solar application. The system was field tested at Queen’s University resulting in a full-scale cooling system unique in North America. The experience of the research team involved prototype solar liquid desiccant air cooling (LDAC) system installations contributed to two large solar cooling installations in Ontario.Footnote 102 The demonstration project contributed to a spin-off project with the Greenhouse Growers of Ontario that used the research and applied it to natural gas in a greenhouse setting.

In the commercial sector, R&D work on carbon dioxide (CO2) secondary loop refrigerant has led to a new Loblaw’s Superstore successfully using the technology and Sobeys has adopted CO2 as the sole refrigerant for all of its new and major retrofit stores.

While there are many examples of progress, not all activities within the Next Generation sub-programs proceeded as expected (e.g., some displacement ventilation / hybrid ventilation field tests; delays in LED lighting installations; ground source heat pump research, delays in installation of Eco-Cute CO2 heat pump; prototype combined fireplace water heater). Common challenges related to difficulties in securing/ maintaining partner engagement; delays in the readiness of field test sites; and loss of capacity in some government departments/agencies.

The Realization of Longer-Term Outcomes

Case study evidence provides examples of achievement over a fairly long time horizon. This is significant because it shows that sustained research may be necessary for achievement of longer-term outcomes. For example, the case study of Zoned System Development highlights how research can progress from the concept stage to the development and deployment of a technology.

As indicated in Figure 5, CanmetENERGY began research on Advanced Integrated Mechanical Systems in the early 2000s. That led to a variety of spin-off concepts, one of which included zoned integrated air handlers.Footnote 103 NRCan had identified a problem (inefficient distribution of heating and cooling particularly in narrow, multi-unit residences) and came up with the concept of a packaged zoned system for new construction. In 2004Footnote 104 the Canadian Centre for Housing Technology (CCHT) examined the potential impact of forced air zoning on various parameters, such as on cooling energy savings. NRCan then solicited calls for proposals to develop and validate the concept of zoned air handlers. The company Ecologix Heating Technologies successfully responded with a product concept. Ecologix then developed zoned air handlers aimed at minimizing cooling loads during summer peak periods. In addition, through another project – Technology Adoption Pilot (TAP) Homes – technologies such as the zoned system were selected through the Local Energy Efficiency Partnerships (LEEP) process and demonstrated in 40 demonstration homes. A large building company in Ontario is building a 117 home sub-division which includes an upgrade option for zoned comfort systems. External stakeholders viewed the leadership role of CanmetENERGY external stakeholders as critical to the success of the system.

Figure 5: Timeline: Zoned System Development (Case Study)

Figure 5: Timeline: Zoned System Development (Case Study)

Text version

Figure 5: Timeline: Zoned System Development (Case Study)

Figure 5 is a timeline graphic, from 2000 to 2011, showing the key developments leading to zoned system development. In the early 2000s CanmetENERGY research identified a problem of inefficient distribution of heating and cooling. As a result the Canadian Centre for Housing Technologies (CCHT) began testing (in 2004) a decentralized approach to cooling that showed potential in terms of energy savings. In 2004-2006 the company Ecologix began the development of the Zone Comfort System in response to an NRCan Request for Proposals. The company also received Industrial Research Assistance Program (IRAP) funding. In 2006 to 2008 market research was conducted (supported through NRCan funding) by the University of Ottawa pertaining to the feasibility of zoned systems for heating and cooling. During this time there were early prototype field trials. Finally in 2008 to 2011 there were field trails with a later stage prototype to quantify energy savings and comfort, and load shifting. There was also a simulation analysis of utility control. The Zone Comfort System was a technology chosen by builders as part of the LEEP project.

3.2.1.4 Enhanced Economic Opportunities

BEP has the potential to enhance economic opportunities by decreasing market barriers through its contribution to codes and standards; through increased industry capacity and by developing more energy-efficient, cost-effective technologies and processes.

As mentioned previously, BEP has contributed to the harmonization of codes and standards through its technical input into the National Energy Code, CSA standards, and international standards organizations.

Many industry partners noted that their involvement in BEP increased their capacity and their visibility which in turn allowed them to market and promote their work. In some instances industry interviewees noted that their participation in BEP increased demand for their work. For example, one industrial partner who participated in the Drake Solar Landing Project estimated that involvement in the project resulted in approximately $6.25M in additional sales in the past few years, with increased sales expected to continue. Another industry partner noted that BEP product testing led to installation improvements and knowledge as to new applications for the product. This resulted in increased sales volume, as well as decreased costs in production of vacuum insulated panels.

During the evaluation timeframe 4 technologies/ systems were commercialized: Phase Change Material; DABO; zoned comfort systems; and solar canopy system. A building envelope technology called Advanced Phase Change Material was transferred to a licensee, Envision Global of Edmonton in December 2009. The company has advanced to product packaging stage under the name “Sunphase Direct Heat Storage Solar Panels”.

Another example of commercialization is the Diagnostic Agent for Building Operation (DABO), a building operational management tool that detects and diagnoses problems and thereby minimizes occupant complaints and reduces energy consumption. It is designed to perform real-time fault detection, flags abnormal component behavior and operational anomalies. DABO® users include: facility operation managers, operators of heating ventilation and air conditioning (HVAC) systems, heads of maintenance teams, and ongoing commissioning professionals.”Footnote 105

DABO is also part of the LowCarbon Initiative, aimed at reducing GHG emissions of NRCan buildings, and was installed in 7 NRCan buildings with installations continuing beyond the evaluation timeframe. A licensing agreement was signed in 2010 between the federal government and IFCS Inc., a company that develops and commercializes building asset management software. In terms of benefits of DABO, case study evidence indicates that it decreases overall operating costs (i.e., 25%) through energy savings while maintaining occupant comfort level. DABO also identified faults in the building electromechanical systems that would have been difficult to detect otherwise. Moreover, DABO facilitated building operator decisions regarding retrofitting and ongoing commissioning. A total of 30 licenses have been sold, in many cases covering multiple buildings in complexes such as universities or hospitals.Footnote 106 Some interviewees noted that sales are not yet as high as expected. Interviewees indicated that progress towards broader deployment may be hindered because there is not a clear economic incentive for some building owners to implement DABO in those cases where owners contract out building operations and pass energy costs onto tenants. As well, it was noted that there are limited building operation resources placing constraints on time available for training and implementation of DABO.

In the area of solar canopy development or core sunlight system, work that began at the University of British Columbia has now spun out to a Canadian company called Sun Central, which is in the process of commercializing the technology. To date there are six installations of this technology. While the Solar Canopy project received the majority of funding from other sources in addition to BEP funding, expertise from NRC and NRCan as well as support from NRCan during the earlier stages of research (NRCan provided funding for proof of concept of the first-generation Solar Canopy Illumination System prototype) was viewed as playing a “small, but significant part” in the commercialization of the product.

3.2.1.5 Environmental Impacts: Energy Efficiency, Energy Intensity, and GHG Emissions

The evaluation could not determine the extent to which BEP R&D has contributed to enhanced energy efficiency and decreased GHG emissions given the myriad of factors that influence this outcome. International literature does recognize the contribution of R&D to reduced energy consumption. Technological innovation can also play a large role in reducing energy consumption of buildings. The energy efficiency of insulation materials, heating systems and other appliances has greatly improved over the past decades and recent developments in solar boilers, geothermal energy and lighting technologies have promise.Footnote 107

Potential for achievement is indicated by the development of energy efficient and cost-effective technologies:

  • The solar liquid desiccant air cooling system was found to provide between 9.2kW and 17.2kW of cooling power with an overall thermal Coefficient of Performance (COP) of 0.40 and electrical COP of 2.43. The collector efficiency was 53%, and 40% of the required thermal energy was provided by the solar array.
  • The Drake Landing Solar Community Project in Okotoks, in the fifth year of the program the project was successful in delivering 97 percent of the community heating with solar energy which was a world record and as a result the project received a 2011 Globe Energy World Award. It was also estimated that five tonnes of GHG emissions were reduced annually per home, a total of 260 tonnes annually for the entire community.Footnote 108
  • With the support of ecoEII funding, testing of the prototype residential baseload monitoring and management system revealed overall power savings of at least 13%.Footnote 109
  • Prototyping, testing and demonstration of high-efficiency self-powered heating and micro electricity generation appliances showed energy savings of over 35%, reduced electricity consumption and GHG emissions.Footnote 110
  • The solar canopy prototype has consistently shown an energy savings of 58% with peak demand energy savings averaging 85%.
  • The MTL showed greater than 70% annual energy savings when compared to being built to Model National Energy Code for Buildings (MNECB) and a 90% savings in space heating energy costs.
  • Testing of cold climate mini split heat pumps shows 60% energy savings possible in zoned operation for both heating and cooling.Footnote 111
  • DABO decreased overall operating costs of buildings (i.e., 25%) through significant energy saving while maintaining occupant comfort levels.

While BEP longer-term environmental outcomes cannot be uniquely attributed to BEP R&D activities, there is an argument to be made that the R&D work has the potential to indirectly contribute to increased energy efficiency and reductions in GHG emissions through its longstanding technical input (particularly on the housing and equipment side) into codes, standards and energy efficiency programs.

Energy Use Trends

Energy use for both residential and the commercial/institutional sectors declined between 2008 and 2011. Specifically, residential energy use declined between 2008 and 2011 by 3.4%.Footnote 112 In the commercial and institutional sectors energy use declined by 2% from 2008 to 2011.Footnote 113

Long term trend analysis (1990 to 2010) shows that the overall energy use in the residential sector increased by 6%.Footnote 114 The rise in the number of households, combined with increased average living space and higher penetration rate of appliances contributed to the increase in energy use seen in the residential sector during this time period.Footnote 115

The long term trend shows that energy use in the commercial and institutional sector increased by 22% between 1990 and 2010. Footnote 116 In the commercial and institutional building sector, space heating accounts for the largest share (45%) of energy use, followed by auxiliary equipment such as computers, printers, and other personal electronic devices which accounts for 19% of all energy consumed. The 41% growth in floor space combined with an increase in the use of auxiliary equipment contributed to the increased energy use in this sector between 1990 and 2010.Footnote 117

GHG Emission Trends

Between 1990 and 2010 (latest year data is available), GHG emissions in the residential sector decreased 0.5 percent as homeowners gradually switched to cleaner energy sources.Footnote 118 While there was an increase in the use of natural gas and electricity during this time, the efficiency ratings of gas and electric furnaces also increased.Footnote 119 For the commercial/institutional sector GHG emissions increased by 15 percent over this period.Footnote 120

Energy Intensity Trends

The observed longer-term increases in energy consumption in both residential and commercial/institutional sectors were offset by improvements in energy use per unit of floor space (i.e., energy intensity). In particular, the residential sector experienced a decline of 29.4 percent in the amount of energy used per square metre between 1990 and 2010, despite the overall increase in the number of household appliances, increased living space and use of space cooling.Footnote 121

Energy EfficiencyFootnote 122 Trends

During the 1990 to 2010 time period there were improvements in the energy efficiency of major household appliances (clothes washers, refrigerators) attributable to the introduction of minimum efficiency standards in the 1990’s.Footnote 123 Specifically, despite a 48% increase in the number of major appliances used between 1990 and 2010, the total amount of energy used to power major appliances during that time period decreased by 24% due to energy efficiency improvements. However, at the same time energy use for minor household appliances (e.g., personal computers, televisions) more than doubled, out-weighing the energy savings achieved for major appliances.Footnote 124

Energy efficiency improved by 36% in the residential sector between 1990 and 2010. These improvements included changes to the residential thermal envelope (insulation, windows, etc.) and changes to the efficiency of energy-consuming items in the home, such as furnaces, appliances, lighting and air conditioning.Footnote 125

Energy efficiency improvements in the commercial/ institutional sector were very similar to those in the residential sector. They included changes to the thermal envelope of buildings (insulation, windows, etc.) and increased efficiency of various energy-consuming items

in these buildings, such as furnaces, auxiliary equipment and lighting. The estimated energy efficiency improvements resulted in a 256 PJ energy savings for this sector between 1990 and 2010.Footnote 126

3.2.2 Unintended Outcomes

Evaluation Question Methodologies Assessment
5. Have there been any unintended (positive or negative) outcomes? Interviews
Document Review
Case Studies
No evidence to indicate any significant unintended outcomes.

Summary:

There is no evidence of significant unintended outcomes arising from the Portfolio activities.

Discussion and Analysis

3.2.3 Economy and Efficiency

Evaluation Question Methodologies Assessment
6. Are the Programs the most economic and efficient means of achieving outputs and progress towards outcomes? Document Review
Literature Review
Interviews
Case studies
There are indications that BEP is efficient and economic with some opportunities for improvement.

Summary:

There are indications that BEP is a well-managed Portfolio, with some opportunities for further improvement. Documents, interviews and case studies provide evidence of efficient and economic practices related to leveraging of resources; the production of high quality outputs; appropriate design; and improved planning and management practices. These lines of evidence also identify areas where improvements could be made relating to internal communication, project selection and review, performance measurement and reporting, and dissemination of research results.

Discussion and Analysis

Planning and Portfolio Design

The strategic planning process for the 2008-09 to 2011-12 cycle was seen by those interviewees involved as an improvement to previous efforts. The strategic plan identified the priorities that linked to the portfolio framework. Several noted that the plan created a framework conducive to the development of more cohesive, targeted proposals at the sub-program level.

In 2010-11, the membership of the Built Environment Portfolio Committee was broadened to include end-users and additional stakeholders such as OEE representatives. The enhanced role of OEE in planning process via their membership on the BEP committee was viewed as a positive step by interviewees to increase take-up of results. However, some interviewees indicated that planning linkages could be further strengthened through enhanced coordination (e.g., sharing of OEE and BEP work-plans to better align work). The need for a strengthened role for other government departments in the BEP committee was cited by some federal interviewees.

While there are some mechanisms in place for seeking external stakeholder input into the planning process, some interviewees suggested this process should be more systematic. Project leads informally seek input from stakeholders prior to planning and submission of proposals. More formal mechanisms include Roadmaps that have been conducted through BEP support or expertise. During the timeframe of the evaluation three roadmaps were conducted which involved soliciting industry, expert and government feedback:

  • Integrated Community Energy Solutions – A Roadmap for Action.
  • Industry Steering Committee for the Sustainable Housing Technology Roadmap. Housing for a Changing World: A Sustainable Housing Technology Roadmap for Canada.
  • Canadian Institute of Plumbing and Heating National Water Heater Roadmap.

As a result of the planning process BEP program structure transitioned from being organized along sector lines (housing, buildings, communities) to technology areas resulting in the creation of three “integration sub-programs” (namely Net-Zero Energy Housing; Net-Zero Energy Buildings and Community PINCH) and a group of seven “next generation technology sub-programs”.

Overall the evaluation evidence suggests that while some streamlining of projects and sub-programs may be necessary to increase potential impact, a broad spectrum approach should remain. The majority of internal interviewees indicate that a broad program (inclusive of integration and a variety of technologies) is preferable given the unpredictable nature of R&D and the changing context in which it operates. As well, both internal and external interviewees indicate the need for integration-focused R&D; an approach which is appropriate for an expansive design.

Many internal interviewees noted that while BEP’s broad scope was aligned with needs and should be maintained, it still required some modest streamlining of projects and sub-programs.Footnote 127 NRCan interviewees suggested that the number of projects could be reduced through enhanced planning, project selection and review.

Some of the international literature on the focus of R&D research indicates this requires a fine balancing act. An OECD/IEA paper (2011)Footnote 128 recommends that an overall energy R&D strategy use a portfolio approach to encompass various technologies at different stages of development and with large-scale potential. While this broad-spectrum approach helps to stimulate a range of promising competing technologies, there is also a danger that spreading funding too thinly across small, sub-critical areas will not produce any long-term benefits. However, international literature also points out that a more selective approach (i.e., picking winners) locks in technologies that may not be economically viable and indicated that the selective approach to R&D may not be an appropriate one for government.Footnote 129

Project Selection

Generally interviewees indicated satisfaction with the project selection process for C-based funding (e.g., ecoETI, CEF, ecoEII). With respect to the PERD project selection process, it was noted that it had undergone changes commencing during the 2008-09 planning cycle with a view to enhancing collaboration among federal researchers.

However, documentFootnote 130 and interview evidence suggested the need for further enhancements of the PERD project selection process. There is a perception amongst OGD and NRCan stakeholders that project selection requires closer alignment with science-based criteria and priorities.

Some interviewees also noted that in addition to science-based criteria, deployment/ marketing factors should be considered when selecting more advanced (i.e., in the development stage) research projects. It should be noted that because PERD provides support for R&D, including earlier stage research, consideration of marketing factors may not be appropriate. However, as some interviewees note, project applications involving more advanced technologies may warrant additional consideration regarding potential for deployment.

Administrative Issues

Interviewees raised concern over the large amount of time spent on responding to calls for proposals for multiple funding envelopes mid-cycle, as well as the tight time frame in which to respond. Researchers note that this administrative burden results in the subsequent loss of time available to conduct research. The tight timeframe is also not optimal regarding the quality of the proposal and the subsequent research work.

Leveraging of Resources

Between 2008-9 and 2011-12 the BEP received $38.8M from PERD, ecoETI, CEF and ecoEII and leveraged $55.4M in cash and in-kind contributions from other sources. As indicated in Table 6, the leveraging ratio of BEP funding (includes PERD, ecoETI, CEF, and ecoEII) to other sources of funding (includes funding and in-kind from a-base and other NRCan, OGDs, provinces, industry, academia, NGO and international sources) was 1:1.4. At the funding envelope level, PERD had the highest leveraging ratio of 1: 2.2.

Table 6: Leveraging Ratios of BEP Funding Envelopes
  PERD ecoETI CEF ecoEII BEP
Envelope funding to other funding 1 : 2.2 1 : 0.58 1: 0.60 1 : 0.02 1 : 1.4
Envelope funding to Non-GOC funding 1 : 1.3 1: 0.4 1 : 0.2 1: 0.02 1 : 0.86

The BEP leveraging ratio of 1:1.4 is comparable to the leveraging ratio of 1:1.5 calculated for the Clean Transportation Systems Portfolio over a similar time frame (i.e., 2007-08 to 2011-12). Footnote 131 The BEP ratio is slightly lower than that of the previous cycle of the BEPFootnote 132 (2004-5 to 2007-8), which was estimated at almost 1 to 2.

Finally, the leveraging ratio of Government of Canada to non-Government of Canada funding for the BEP was 1 to 0.55. This is slightly lower than ratios calculated for other Clean Energy S&T Portfolios in the previous cycle such as for Clean Electric Power Generation (CEPG) (2003-04 to 2008-09) which was 1:0.79Footnote 133 and for Transportation S&T (2002-3 to 2006-7) which was 1:0.62.Footnote 134

Interviewee evidence from both internal and external stakeholders suggests that the ability of BEP researchers to successfully leverage significant PERD funds can be attributed in large part to the good reputation of BEP research. OGD and NRCan representatives noted that PERD provided seed money to leverage other funds and this was supported through case study evidence. However, many NRCan interviewees noted that declining A-base and increasing reliance on PERD funding hinders their ability to leverage other resources.

Planned Delivery of Outputs/Quality of Outputs

A review of performance reports as part of the evaluation indicated that most expected outputs (70%) were delivered as planned. Given the unpredictable nature of R&D it is not surprising that all outputs were not delivered as planned. Determining the adequacy of this percentage is difficult in that there are few benchmarks available for comparison. While not entirely comparable, an evaluation of U.S. DOE R&D projects found that 72% of projects met their technical objectives, lending support to the idea that not all R&D projects can be expected to progress as planned.Footnote 135

Delays were reported for about 30 percent of the R&D work. Reasons cited for the project delays were (in order of decreasing frequency):

  • problems with the testing or demonstration site;
  • partner disengagement;
  • contracting issues;
  • decreased capacity experienced by some government departments/agencies;
  • technical/ equipment issues;
  • workload issues; and
  • time taken to obtain occupant approval to monitor building performance.

Reported delays were more common in research projects involving field trials, which are not surprising given the number of factors, some of which are beyond the control of project management, influencing their implementation.

Program documentation indicates that just over half of the cases of delays resulted in the work being continued into the next funding cycle or in reductions in the scope of work. In some cases, it was noted that funds were redirected to more promising areas and in other cases, there was minimal or little impact cited from the delay.

The end-of-cycle external reviewFootnote 136 rated most sub-programs (seven of the eleven) at 70% or better in terms of delivering planned outputs. This review also concluded that for some of the larger projects, reasons for lack of progress typically included field test/demonstration site issues or changes in other federal department’s priorities. This assessment did not account for additional outputs developed instead of, or in addition to, those planned or for those resources that were redirected to more promising areas, which better facilitated the success of some projects (e.g., LEEP/TAP). It should also be noted that given the nature of R&D, not all projects can be expected to proceed as planned.

Finally, document, case study and interview evidence suggest that in general outputs were of high quality. Most external interviewees contacted for case studies made reference to the high quality and usefulness of the project’s R&D outputs. Many external interviewees within academia, the public, private (industry) and international sectors also mentioned the high quality of research and technical advice occurring through BEP funding, notably in the areas of energy use modelling, lighting, micro-cogeneration, and building preventative maintenance and fault detection.

Communication and Collaboration

In the evaluation reference period there have been increased efforts to improve collaboration and communication among researchers. For example, researchers were required to submit integrated proposals for each sub-program during the 2008-09 and the most recent PERD planning cycles. While the evaluation found some examples of good communication and collaboration particularly within some sub-programs, on the whole there is a need to enhance collaboration among BEP researchers and across sub-programs.

Many interviewees indicated the need for enhanced communication among BEP researchers. While some indicated the current measures (i.e., semi-annual meetings at the sub-program level, the brief status updates, and the annual end of year workshops at the Portfolio level) were adequate for sharing results, networking and increasing awareness, many other interviewees suggested more could be done to facilitate more meaningful interaction at these sessions or through other mechanisms.

Interview evidence indicates that communication between BEP researchers and OEE housing and equipment representatives is strong, but that there is a need for improved communication between BEP researchers and OEE building representatives.Footnote 137 The stronger communication between BEP and OEE housing and equipment representatives is likely due to the longstanding nature of these partnerships.

There were some examples of collaboration among government departments and agencies involved in BEP. For example, between CMHC and NRCan (e.g., monitoring of demonstration homes for the Equilibrium Housing Initiative). Moreover, the CCHT provides a good mechanism for communication and collaboration among NRCan, NRC and CMHC. Other examples include Solar Canopy work that involved PWGSC, NRCan and NRC. However, interview and document evidence indicated that overall collaboration among federal researchers should be improved both within sub-programs and across sub-programs.

Improved coordination across sub-programs is needed. A review of PERD/ecoETI progress reports show coordinated efforts for some research projects (e.g., Kortright Sustainable Archetype house, air distribution design methods, zoned comfort system, climate air source heat pumps). The Sub-programs most frequently cited in progress reports as linking to other sub-program areas of research were NZEH and NG Mechanicals and Renewables sub-programs. Interview and document evidenceFootnote 138 indicates that enhanced coordination and stronger linkages within the NZEB sub-program as well as to appropriate buildings research being conducted in other next generation technologies sub-programs would benefit the research.Footnote 139

Performance Measurement and Reporting

Efficient performance management practices demonstrated during the evaluation period include regular project performance monitoring and reporting of output achievement against milestones, and the conduct of three internal reviews: one Impact AssessmentFootnote 140 of the BEP going back four cycles; one mid-cycle reviewFootnote 141 ; and one end-of-cycle reviewFootnote 142 . As an indication of good management, the reviews, particularly the results of the end-of-cycle review, were linked to the most recent PERD planning cycle that commenced in 2012-13.

Most interviewees noted that the bi-annual frequency of reporting was appropriate. In general, NRCan interviewees also said that the performance monitoring spreadsheet format was adequate to report on outputs, but was cumbersome and not the best format to explain or assess progress of the work or uptake of results.

Analysis of the project reporting templates (spreadsheets) as part of this evaluation revealed considerable variation in the way sub-programs reported on project progress in the spreadsheets, especially in terms of describing delays and changes in research direction, and contribution to outcomes. Performance monitoring does not consistently report on uptake or link project outputs to portfolio outcomes. This situation makes it difficult to aggregate results to the portfolio level. The variability in performance reporting impacts the interpretation of a project’s/sub-program’s success by reviewers, and makes it difficult to roll up and assess progress at the portfolio level. Some senior management representatives also noted that it is difficult to know the “portfolio performance story”. Moreover, more consistent and systematic performance reporting would enable decision-makers to periodically review the status of projects and make informed decisions as to whether to cancel the project and redirect resources to more promising areas.

Suggestions to improve performance reporting include:

  • improve consistency of reporting outputs and outcomes through more structured reporting templatesFootnote 143 ;
  • systematically report on changes in research direction; reasons and impact of any delays; and on transfer and uptake of research outputs; and
  • regularly consult with partners/end-users to assess transfer and take-up.

Dissemination of Results

While there are many good dissemination practices noted by the evaluation, there was overall agreement among NRCan interviewees as to the need to improve dissemination. Insufficient resources, travel restrictions, length of time taken for approval to publish results, and limitations of publishing on the federal government websites in terms of space, formatting, and accessibility were cited as key barriers to dissemination.

Interviewees provided several suggestions to improve dissemination such as:

  • Create a designated position within BEP to coordinate dissemination activities.
  • Create a repository to increase access to existing information and research results. Ensure products are tailored appropriately for different target audiences, especially non-academics.
  • Enhance visibility and publicity of work by using: multiple formats; social media; workshops; electronic mailing lists; emails with clickable headlines; summaries/ research highlight reports; obvious links on websites; more demonstrations and include signage with a government logo (several external stakeholders highlighted the credibility that goes with the federal government and that a simple sign with federal department’s logo at a demonstration was considered valuable and effective).
  • Consistently hold wrap-up meetings with project partners and stakeholders to present research results.
  • Strengthen linkages and communications with industry associations (particularly with respect to buildings and communities) to champion and disseminate research results.
  • Establish forums/roundtables to share technical results to broader industry.

Best Practices

Several best practices have been identified primarily through the case studies. The most successful projects tended to have some common best practices such as the early and ongoing engagement of stakeholders; clear and regular communications with project partners; and employ strategies /incentives to maintain partner buy-in. Examples of good practices demonstrated or learned through BEP research projects include:

  • Effective early engagement of key stakeholders in the planning process:
    • For example, a challenge to technology demonstration is the time it often takes to secure approvals for new materials, designs or construction techniques. It is important, therefore, to include building inspectors and city building officials in project design, delivery and outreach activities.
  • Maintaining partner buy-in and engagement:
    • The consortium method used by NRC is viewed as an effective means to engage key stakeholders throughout the R&D process and involves members sharing costs and results of the research. This is supported by the literature in which an OECD/IEAFootnote 144 paper emphasizes involving industry early and keeping them onboard throughout to ensure relevance.
    • Enhance partner engagement through research that provides useful information that may not be directly related to energy efficiency goals of NRCan, but nevertheless contributes towards those goals. For example, provide market analysis and information to small companies to facilitate commercialization. Inclusion of human behavior elements (e.g., comfort) is of interest to many external stakeholders as they can have marketing and practical implications.
    • Provide training/ presentations to key groups to ease potential barriers to project implementation. In the case of the LEEP/TAP project, including municipal building officials in workshops facilitated the issuing of permits.
    • Encouraging buy-in of building owner partners was facilitated in the Post –Occupancy Evaluation Project by providing individual confidential building results to each owner and preserving the anonymity of those results.
    • In the case of the Drake Landing Solar Project, Drake Landing is owned and operated by a group of private businesses and the Town of Okotoks which gives partners a vested interest in ensuring the project succeeds.
  • Effective technology deployment was demonstrated in the LEEP/TAP project, a builder-led technology assessment, selection and field trial process. LEEP is delivered through a series of facilitated workshops in which builders can review and select innovative technologies of interest to them. LEEP was augmented by adding a field trial component called the Technology Adoption Pilot (TAP). Case study evidence highlighted the effectiveness of this process in deploying energy efficient technologies.
  • Effective project management:
    • The Solid State Lighting case study illustrated how the project built on lessons-learned during the introduction of compact fluorescent light bulbs (CFLs).
    • Regular and frequent communication with partners during the project is important. Case studies show that regular updates on project progress and meetings contributed to the identification of issues and their resolution.
    • Take advantage of the flexibility of external funding. In one case study (Post Occupancy Evaluation) external funding was used primarily in the later years, which were the most expensive due to the field work.
  • Effective dissemination strategies:
    • Some interviews noted that in the past, there have been annual seminar series highlighting R&D through various themes. Interviewees said this was an expensive but effective method of reaching practitioners. Some interviewees noted that something similar could be done electronically.
    • Provide summaries of research activities that are customized to the target audience to facilitate stakeholder / partner engagement.
    • Disseminate information through a variety of means. For example, convene one or two post project web-conferences with the partners to discuss the application of results.
  • Considerations for successful technology development/demonstration projects:
    • Homeowners need a system that can be simply operated and requires little maintenance.
    • In depth front-end work helps to ensure technology is sound and motivates the private sector to become involved. For larger demonstration projects, ensure that all major equipment is tested before installation and assess its overall impact on the system.
    • When addressing innovation and R&D needs in the North, consider the context in which the projects are taking place. Innovation should include implementing existing, proven, technologies in novel, northern housing designs.

3.2.4 Factors Affecting Performance

Evaluation Question Methodologies Assessment
7. What are the internal and external factors influencing the effectiveness, efficiency, and economy of the programs and activities? Document Review
Literature Review
Interviews
Case Studies
Evidence indicates that there are several internal and external factors that can facilitate or hinder success of projects and programs of research.

Summary:

Key external factors influencing the effectiveness, efficiency, and economy of BEP are energy prices, cost of constructing /retrofitting net-zero houses and buildings, access to energy data, complexity of the building and community sector, and provincial environmental policies and programs. Internal factors that influence BEP include federal government expertise and leadership, the multi-year PERD funding structure, a declining NRCan A-base, and accessing multiple funding envelopes.

Demonstration funding programs such as TEAM and deployment programs such as those managed by NRCan’s OEE have been substantially reduced, which negatively impacts dissemination and knowledge transfer.

Discussion and Analysis

Interview, document and case study evidence point to a number of internal and external factors that can positively or negatively influence the effectiveness, efficiency, and economy of the programs and activities. The key message that can be derived from the international literature on R&D is that no one policy instrument can likely work in isolation – a broad range of policies and measures are appropriate to deal with GHG emissions.Footnote 145

The internal and external factors influencing BEP are discussed below.

Internal Facilitating Factors

  • Federal government expertise and leadership was identified in most case studies and by many external stakeholder interviews as critical to the success of the projects.
  • The multi-year, ongoing PERD funding facilitates longer-term planning, and the continuity of research. C-based funds such as ecoETI and CEF are used to complement and build upon PERD research to bring them farther along the innovation continuum.
  • Strong sub-program coordinators with good communication and project management skills were seen by many BEP delivery staff as important to the achievement of results.

External Facilitating Factors

  • Momentum in the green building sector. NRCan interviewees believe that there has been an immense increase in interest across the country for low carbon energy delivery, and that labelling programs like LEED certification, incentive programs and supportive provincial and municipal environments are providing opportunities and interest for energy efficiency improvements.
  • Higher energy prices in the North /and in some provinces. Higher energy prices in the North create more opportunities for the application of BEP R&D. For example, a BEP project on solar seasonal storage technologies for net-zero community and building applicationsFootnote 146 concluded that these technologies are considered to be of significant interest to utilities only in regions of Canada where heating costs are above $30 per gigajoule, such as the Yukon and Nova Scotia.Footnote 147
  • Supportive Provincial Energy Conservation, Climate Impact Mitigation Strategies. For example, in British Columbia, the legislative and policy environment is supportive of municipalities reducing energy use and GHG emissions. The 2008 Local Government (Green Communities) Statutes Amendment Act (Bill 27, 2008) introduced changes that require local governments to include GHG reduction targets, policies and plans for action in their Official Community Plans by 2010, and to set energy requirements for new developments.Footnote 148 An official community plan is a local government by-law that establishes objectives and policies to guide decisions on planning and land use management within its jurisdictional boundary. Footnote 149 In general, provincial and territorial legislation and policies do not require the consideration of energy planning in community planning processes.Footnote 150

Internal Hindering Factors

  • Declining NRCan A-base. There was strong agreement among internal interviewees that the declining A-base and increasing reliance on PERD to fund salaries decreased incentives to collaborate with other federal government stakeholders. Some interviewees indicated that this also has negative implications for PERD’s capacity to leverage other sources of funds.
  • Insufficient internal capacity and resources to conduct outreach and dissemination was identified by both internal and external stakeholders as a key factor hindering awareness and up-take of research results.
  • Changing rules to Government of Canada’s web content; the effort required to meet accessibility and translation requirements prevents timely and fullFootnote 151 publication of materials.
  • Inadequate Mechanisms for Industry and Federal Government Collaboration. Some NRCan interviewees noted inadequate mechanisms (e.g., contribution agreements, contracting) for collaboration between the private sector and federal researchers. Both internal and external stakeholders noted the mutual benefits of collaboration (for both R&D and for demonstration projects) in providing important contributions to policy and in helping build the capacity of the housing and building sectors.

External Hindering Factors

  • Low energy costs. The external factor most frequently cited by internal and external interviewees was the relatively low price of conventional energy sources, especially natural gas. Between 2008 and the beginning of 2013, the price of natural gas declined by approximately 60% percent.Footnote 152 According to the National Energy Board, the lowest natural gas prices in over a decade were witnessed in 2012, with Intra-Alberta gas trading below $2.00 per GJ from March to June.Footnote 153 Low energy prices make alternative energy technologies less economically feasible, reducing economic incentives to be energy efficient.
  • High up-front cost of construction /retrofitting of net-zero houses and buildings. There is overall agreement among internal and external interviewees that one of the key barriers to uptake of net-zero houses and buildings are high up-front costs. According to a 2011 survey of homebuilders, cost is the greatest barrier to building net-zero energy homes.Footnote 154 For example, one projectFootnote 155 found that due to a lack of economy of scale of the thermally driven chillers, payback periods can be as long as 25 years. Footnote 156 Another projectFootnote 157 concluded that even looking at best-practices in Net-Zero and low energy home design in North America, these homes add approximately $80,000 to the cost of a home.Footnote 158
  • Insufficient demonstration programs/funding. Both internal and external interviewees noted the importance of demonstrations in the built environment to develop industry capacity and mitigate the risks of adoption. While ecoEII provides some demonstration funding relevant to the BEP, there are insufficient demonstration projects targeted to the built environment that would provide opportunities for industry. Additionally, some external stakeholders indicated that there is a particular funding gap for small demonstration projects targeted at small and medium sized enterprises (SMEs).
  • Complexity of building sector. Canada’s commercial building sector is complex and includes a variety of building types, ranging from offices to hospitals and schools. Stakeholder groups are diverse and include investors, builders, engineers and architects, real estate agents, tenants, and building operators.
  • Unclear/split incentives for the uptake of energy efficient technologies in the buildings sector. One of the most frequently cited barriers to energy efficiency in buildings is the split incentive between those who construct or own the building and those who pay the energy bills.Footnote 159 There are often unclear incentives for the uptake of energy efficient technologies in the buildings sector because building owners may not always benefit from an efficiency project as they can often pass energy costs on to their tenants. Furthermore, energy efficiency investments often have to compete against other uses for capital and equity (e.g., meeting payroll requirements or expanding product lines) and thus they may not be adopted.Footnote 160
  • Challenges obtaining timely and good quality data on energy consumption. Many case studies illustrated challenges of obtaining timely and good quality energy use data.
  • Difficulties associated with engagement of some external stakeholders. Case studies, document review and interviews indicate that it can be difficult to engage partners, particularly industry, building owners and operators, and utility partners. Since utilities operate on the basis of revenues, provinces need to be on board and provide energy savings/ conservation targets in order to provide an impetus for utilities to engage in energy efficient initiatives.
  • Loss of OGD Capacity. Changes in future research staffing levels at OGD partner organizations such as CHMC, EC and PWGSC has resulted in a loss of expertise in some areas such as advanced durable wall systems, thermal energy storage, and energy management and operations. In addition, PWGSC is a key potential receptor of building related BEP R&D. However, since PWGSC research capacity was reduced in 2010, there is an increased risk that its receptor capacity has diminished. Similarly, the diminished capacity and shifting priorities of OGDs (CMHC, EC) may weaken their capacity to disseminate and use BEP information for policy and program development.
  • Energy planning at the community level is complex and nascent. Some key barriers to the implementation of integrated community energy solutions identified by interviewees and the Council of Energy MinistersFootnote 161 include: limited awareness of and the technical capacity to do energy mapping; lack of nationally consistent methods for measuring energy at the community level; inter-jurisdictional complexity/multiple sectors involved in decision-making (e.g., energy supply and distribution, transportation, housing and buildings, industry, water and waste management, and land use sectors); conflicting or unclear policy and regulatory environment across jurisdictions; and limited consideration of energy considerations in community planning.

4.0 Conclusions and Recommendations

BEP activities are relevant, consistent with federal priorities and the federal and NRCan roles. BEP R&D is needed to advance built environment technologies and knowledge that have the potential to contribute to GHG emissions reductions and increased energy efficiency.

The Portfolio has performed well in developing knowledge and tools that have made important technical contributions to codes, standards and energy efficiency programs. BEP has largely been effective in achieving immediate and some intermediate outcomes although progress towards longer-term outcomes is inconsistent across sub-programs. Achievement of intermediate outcomes relating to use and deployment of research outputs is more evident among the Net Zero Energy Housing and several next generation technology areas. While the inconsistencies in uptake relate in large part to external factors, there is also some evidence to suggest factors such as enhanced communication and collaboration could facilitate more consistent outcome achievement. A more comprehensive dissemination strategy is required to enhance the effectiveness of the Portfolio as a whole. External stakeholders indicate gaps in funding for demonstration projects, particularly with respect to smaller scale demonstration projects suitable for smaller companies. Declining resources, in particular, the declining NRCan A-base for R&D was a common concern among federal interviewees.

BEP is a well-managed program, with noted improvements in planning and performance monitoring in comparison to the previous planning cycle. However further enhancements to the performance measurement and reporting are needed to provide a more complete understanding of progress, particularly at the sub-program and portfolio levels.

Recommendations

The Evaluation has four recommendations to enhance the relevance and performance of the Portfolio.

Recommendation 1: NRCan continue to identify and strengthen strategies to improve internal communication and coordination among BEP researchers and between BEP and OEE, particularly on the buildings side.

Recommendation 2: NRCan continue to strengthen its current review process to clearly identify which BEP projects are not progressing as planned, and to take appropriate actions.

Recommendation 3: NRCan enhance dissemination of BEP outputs by developing and implementing a more comprehensive dissemination strategy that includes strengthened links to industry associations.

Recommendation 4: NRCan further enhance the BEP performance measurement and reporting strategy by improving the consistency of reporting and making clear links to outcomes. Reporting on outcomes would be facilitated by regularly obtaining feedback from external partners and receptors.

Annex A: Case Study Descriptions

#1. Title: Post-Occupancy Evaluation (POE) Of Green Buildings

Total Project Value (all sources): $896,000 Timeline: 2008-09 to 2011-12

Objective: To demonstrate the value of green building practices in terms of improved indoor environmental quality and decreased energy consumption.

Target Audience: Primarily green building councils; the Building Technology Transfer Forum; and the participating building owners. Secondly, other building owners, organizations interested in green building policies, and industry consultants (e.g., architects).

Federal Lead: National Research Council - Institute for Research in Construction.

Partners: Public Works and Government Services Canada; Consortium members (Governments of: AB, MB, SK, NS, NB and ON; BC Hydro; Fonds en Efficacite Energetique - Gaz Métro; Haworth Ltd; McClung Lighting Research Foundation; and University of Idaho); New Buildings Institute (NBI), USA; U.S. Green Building Council; Canada Green Building Council; Building Owners and Managers Association; Green Globe.

Summary: This project addressed two key questions: i) Do green buildings deliver the higher quality indoor environments and lower energy use that they promise? If not, why not? And, ii) How might rating-schemes be modified to ensure a greater chance of success?

Key outputs of the project were a literature review; supplementary analysis of existing energy use data from more than100 LEED-certified buildings; and post-occupancy evaluations of 12 pairs of operating green and conventional office buildings providing comparative data on the physical indoor environment, energy performance, occupant satisfaction, and factors affecting organizational productivity. The study found that green buildings were performing well in comparison to conventional buildings, but identified areas for improvement. Green buildings had superior indoor environment performance, but while on average the LEED certified buildings studied used between 18-39% less energy per floor area than their conventional building counterpart, approximately a third of the LEED buildings used more energy.

Interviews with external stakeholders indicate that the project increased their awareness and understanding of energy efficient practices and green building performance issues; facilitated strengthened and new collaborations to facilitate the use of green buildings tools and practices; and that they used the research results to make decisions about their green building policy and practice.

Some lessons learned/best practices:

  • Access to consistent building energy use data is challenging.
  • Encourage the participation of building owners by providing individual building results to each owner and preserving the anonymity of those results.
  • Engage stakeholders from planning to final stages of the project to enhance buy-in and ensure the project meets everyone’s needs. In this case a consortium approach was used in which the group shared the costs and the research results.

#2. Title: Improving the Energy Efficiency Of Existing Government Buildings

Total Project Value (all sources): $480,000 Timeline: 2008-09 to 2011-12

Objective: To (a) develop a quick and comprehensive process to estimate energy consumption of office buildings; and (b) to optimize the selection of the most effective energy retrofit measures.

Target Audience: Facility managers; engineers; architects; operational managers; energy managers; and planners.

Federal Lead: Public Works and Government Services Canada (PWGSC)

Partners: McMaster University; International Energy Agency Annex 46; Pacific Northwest National Laboratory; U.S. Department of Energy.

Summary: This project involved developing a model designed to show which energy conservation measures (ECMs) are effective based on energy savings, and which are financially viable within a large stock of new or existing office buildings.

Key outputs of the project were the Office Buildings Energy Prediction Model (OBEPM) and a retrofitting guide on the proper use of energy retrofit/conservation measures. While there are a number of existing tools for modeling energy performance of buildings, the OBEPM is considered unique in that it is a more practical and user friendly tool for deciding among energy retrofit options within the Canadian context (e.g., the model can make comparisons between ECMs based on energy consumption savings for buildings of various sizes, archetypes, and in various locations; and can rank individual ECMs for large stocks of office buildings). Overall, based on comparative assessment of the model against the US EnergyPlus model (considered a very robust tool), the acceptability of the OBEPM model predictions is considered good given its intent as a preliminary evaluation of building energy use. However, development of the model is continuing and as such it has not yet been widely deployed.

Interviews with stakeholders indicate that the project has increased awareness of the model and energy use and efficiency within PWGSC technical groups, university partners and students. The model has been incorporated into graduate courses at McMaster University, and has contributed to the work of the IEA’s Annex 46. Finally, as a result of the professor’s involvement in the project, a utility has approached McMaster for information about similar ECM decision tools for school buildings, thereby enhancing collaboration opportunities.

Some lessons learned/best practices:

  • Federal expertise and ongoing communication with partners was critical to the success of the project.
  • Develop protocols for federal monitoring of building energy consumption and building operations. This will facilitate the development of decision tools regarding energy savings, building operations and energy conservation measures.
  • Senior management support is critical for ensuring that energy consumption data is shared.

#3. Title: A System-Wide Methodology For Optimising Renewable Energy Solutions (SMORES)

Total Project Value (all sources): $954,450 Timeline: Nov. 2010 – Mar. 2012

Objective: To develop a holistic decision-making process/methodology to assess the optimal mix of renewable energy technologies, demand side management activities, and municipal policies for application at a community-wide scale.

Target Audience: Community planners; utilities; and program developers. Secondary targets are policymakers at all levels of government.

Federal Lead: NRCan, CanmetENERGY

Partners: National Research Council; Public Works and Government Services Canada; City of Ottawa; National Capital Commission; Carleton University – Carleton Sustainable Energy Research Centre; Hydro Ottawa; Enbridge Gas Distribution; Technical University of Denmark

Summary: This project aimed to reduce barriers to the extensive deployment of renewable energy systems by developing a decision-support tool to help communities evaluate the technical feasibility and socioeconomic benefits of large scale implementation of renewable energy technologies, policies and programs as part of a community level energy plan. A community in Ottawa was used as the study site.

Key outputs of the project were descriptions of the energy-relevant characteristics of the project test community; identification of options for energy sources within the community; technical and costing specifications for a range of alternative energy technologies for electricity generation, electricity storage, heat generation, distribution and storage; a description of the developed decision-support tool for scoping out the optimal mix of energy supply technologies for their specific circumstances; and a description of the application of the tool in the test community. The project was unable to obtain sufficiently detailed data in the time frame available to complete the original goal of quantifying the impact of integrating energy technologies, policies and programs for optimizing energy efficiency and GHG reductions. Broad dissemination of the project results was hampered by several factors including limited budget, resources and publication protocols.

Despite the data collection and dissemination challenges, partners reported that the project enhanced their research capacity and was a useful validation of the challenges posed by district energy. The project also reportedly resulted in new and strengthened relationships among team members and between an external stakeholder and NRC.

Some lessons learned/best practices:

  • Senior management vision and support is a critical ingredient for success of a project with a long-term vision.
  • Community projects of this nature require a broad skill set beyond energy engineering to include in-depth knowledge of city planning process.
  • Active engagement and vested interest of key players in the project facilitates the success of the project.

#4. Title: Centrally Zoned Forced Air System Development

Total Project Value (all sources): $1,460,000 Timeline: 2008-09 to 2011-12

Objective: To examine the energy reduction potential and comfort implications of centrally zoned forced air systems and of the use of utility controlled thermostats in homes with and without the zoning system.

Target Audience: Builders; electric and gas utilities; the Ontario Power Authority; HVAC manufacturers and contractors and manufacturer/contractor associations; and homeowners.

Federal Lead: NRCan, CanmetENERGY- Ottawa, Housing

Partners: Ecologix; Strack & Associates; McMaster University; Ontario Power Authority; Utility Partners: Chatham-Kent Hydro and Kitchener-Wilmot Hydro.

Summary: The zoning study compared the energy consumption and zone-by-zone indoor temperatures and relative humidity of 15 homes in southern Ontario equipped with the centrally zoned forced air system manufactured by Ecologix (the “Zone Comfort system”) to 5 homes without the system, during both heating and cooling applications. The follow-on project examined the energy and comfort implications of the use of remote thermostats or load control switches during cooling applications in homes with and without the zoned system.

Overall, the study found that residential zoned heating and cooling systems have the potential to reduce annual energy consumption and improve homeowner comfort. Furthermore, the functionality of these systems can be enhanced with programmable thermostats and additional utility control devices to reduce peak summertime electrical loads and shift loads to off-peak times. The researchers note that a larger sample is needed to validate and build on the results.

Niche builders have increased their awareness of zoning technology through dissemination activities of a parallel project (the Local Energy Efficiency Partnerships/ Technology Adoption Pilot project) and through the Canadian Home Builders’ Association’s technical research committee. This in turn has led a Toronto builder (Arista Homes) to install zoned forced air systems with optional smart-grid thermostats in a new 117 home sub-division. Interviews indicate that the project has enhanced a manufacturer’s R&D and marketing capacity; led another manufacturer to develop a competitive product; enhanced research collaborations between the university and utilities; influenced one utility’s thermostat intervention plans; and led NRCan to work with HRAI to standardize zoned distribution design and installation. Finally, in part as a result of the field trials and related market growth, the installed price for zoned systems has been reduced by a factor of 5 over the past 5 years.

Some lessons learned/best practices:

  • Involve key partners in the planning stages to help ensure buy-in during the whole project
  • Conduct wrap-up meetings with partners/receptors using less technical language
  • Enhance partner engagement through research that provides useful information that contributes to, but may not be directly related to, the energy efficiency goals of NRCan
  • Provide market analysis and information to small companies to facilitate commercialization

#5. Title: Drake Landing Solar Demonstration Project Follow-up Activities: Monitoring & Technical Support and Feasibility Study of a Larger Scale Solar Demonstration Project.

Total Project Value (all sources): $3,124,000 Timeline: 2008-09 to 2011-12

Objective: To monitor operational and energy performance of the Drake Landing Solar Community and optimize the system; to provide technical and operational support; and to determine the technical and economic feasibility of a larger scale solar community project.

Target Audience: Regulators; utilities; planners; land developers; architects and home builders; energy systems professionals; and the thermal energy storage research community.

Federal Lead: NRCan, CanmetENERGY-Ottawa, Active Solar

Partners: Town of Okotoks, AB; Sterling Homes, AB; ATCO Gas, AB; Enermodal Engineering, ON; SAIC Canada; École Polytechnique, QC; EnerWorks Inc.

Summary: A key research activity was to monitor and optimize the performance of Drake Landing, a 52-home demonstration solar seasonal energy storage project in Okotoks, Alberta completed in 2007-08. Optimization of the system resulted in improved performance, achieving a world-record breaking solar fraction for community space heating requirements of 97% in year five of operation, and in a calibrated and improved simulation design model. The feasibility study found that for a 1000+ community in southern Alberta, the solar seasonal storage technology overall costs could be reduced by about 53% as compared to the Drake Landing project. The feasibility study noted that the economic feasibility is highly dependent on the regional supply and cost of space heating energy.

External partners increased their capacity and expertise in solar / community energy technologies, and partners and owners of the Drake Landing Solar Company reported that the project’s participative approach enhanced their relationships. Partners and receptors are using knowledge acquired through the project to, for example, modify or develop a technology/ practice/ or similar project; conduct research; and lower business costs. Gains in the solar fraction achieved (90% to 97%, a world record) translate into an estimated 260 tonne reduction in annual community- level GHG emissions. Finally, participation in the project led to increased sales / projects for some partners; one company attributed approximately $6.25M worth of sales to enhanced capacity and marketability gained from involvement in the project. The project received the Energy Globe Foundation Golden Energy Globe World Award in 2011.

Some lessons learned/best practices:

  • Implement a monitoring system on which to judge performance and make adjustments.
  • Regular and frequent communication with partners contributes to the identification of issues and their resolution.
  • In depth front-end work (e.g. review of best practices in Europe) helps to ensure technology is sound and motivates private sector to become involved.
  • Have the right partners, with appropriate expertise and motivation (e.g., planners, owners)
  • Homeowners need a system that can be simply operated and requires little maintenance.
  • Test all major equipment before installation and assess its overall impact on the system.

#6. Title: Diagnostic Agent For Building Operation (DABO)

Total Project Value (all sources): $1,008,000 Timeline: 2008-09 to 2011-12

Objective: To enhance several components of DABO; develop an energy management and prediction tools; and develop a dashboard that provides hourly summaries of DABO analysis.

Target Audience: Building/facility managers; control system technicians; building operators; utilities and commissioning providers.

Federal Lead: NRCan, CanmetENERGY- Varennes

Partners: Environment Canada; PROFAC; Palais des congrès; Centre universitaire de santé McGill; Hydro-Québec; Cofely Services; IFCS Inc.; SNC Lavalin.

Summary: DABO is an information-based, ongoing commissioning tool that is designed to ensure the continuous optimum operation of buildings and their electromechanical systems. Research contributing to the development of DABO began at CANMET-Varennes in 2000.

Key project outputs consisted of a new energy management component for DABO that uses building control data to predict building energy demand; and an ongoing commissioning component which provides continuous feedback on a building’s performance and energy consumption, and identifies opportunities for improving building operations. The project involved several field demonstrations and initiated the commercialization process.

External stakeholders reported that DABO contributed to an increased awareness of energy efficient building operation. DABO was used on a demonstration basis to operate buildings and make decisions about retrofitting and ongoing commissioning; has been integrated into NRCan’s Low Carbon Initiative as a tool for diagnosing the operations of NRCan buildings and is installed at all NRCan forestry centres. Demonstration participants noted that DABO was useful in detecting and diagnosing building operation faults that would have been difficult to detect otherwise and reported decreases in operating costs (i.e., 25%) through energy savings while maintaining occupant comfort level. To date, DABO has been installed in China, Belgium, France, Australia and Canada. A licensing agreement was signed in 2010 between the federal government and IFCS Inc., and a total of 30 licenses have been sold in Canada, in many cases covering multiple buildings in complexes such as universities or hospitals.

Some lessons learned/best practices:

  • Multi-disciplinary approach
  • Support/training provided by CanmetENERGY was very helpful during demonstration projects
  • To ease the burden on limited building operation resources, introduce flexibility in the selection of DABO logical rules to ensure warning messages are relevant to users

#7. Title: Optimization Of Domestic Hot Water Systems

Total Project Value (all sources): $576,400 Timeline: 2008-09 to 2011-12

Objective: To develop a representative draw profile that reflects current Canadian household hot water consumption; and to assess the energy efficiency of alternative, advanced domestic hot water (DHW) technologies and configurations.

Target Audience: NRCan OEE; Canadian Standards Association (CSA); utilities; Canadian Gas Association; manufacturers/distributors of DHW systems and related equipment; Canadian consumers.

Federal Lead: NRCan-Innovation and Energy Technology Sector (IETS)

Partners: Office of Energy Efficiency (OEE), NRCan; manufacturers and distributors of domestic hot water systems; Gas utilities (Ontario).

Summary: The project had two key activities. One was to conduct a field study of residential hot water use to update the previous study completed in 1985. Water use profiles are used in the development of hot water heater test methods and standards. The second activity involved the design, development and performance testing of four advanced DHW systems. Results of this work will help industry meet the new, more stringent minimum energy efficiency standards for gas and electric water heaters planned for 2020. According to interview findings and project documentation, several outputs were delayed due to technical issues and insufficient lab access, including the assessment on one of the four advanced systems.

Key outcomes include improved data and research on advanced DHW technologies, revised CSA P.3 and P4.1 Standards, and a greater understanding and awareness among industry, standards groups, and consumers of the potential to cost-effectively reduce water heating energy use through the use of appropriate, advanced technologies. Research findings showed that the current habits and water use are significantly different from the assumptions used 25 years ago to develop the CSA P.3 performance test standards that are still in use today. The updated domestic hot water use profile developed in this project are being incorporated into the 2013 revision of the CSA P.3 Standard, which will provide more accurate estimates of the energy efficiency of existing and new DHW systems. Also, research on one of the DHW advanced technologies led to a revision of the CSA P.4.1 (Gas Fireplace Performance Standard) to include provisions for evaluating a condensing gas fireplace.

Some lessons learned/best practices:

  • Changing the P.3 test standard to better reflect ‘real life’ water use patterns will result in very different efficiency assessments for DHW technologies than if using the old standard.
  • Promoting the development and uptake of advanced DHW systems in Canada will require more outreach to, and support for, manufacturers who have little to no experience with these systems. A best practice to support industry is the ‘manufacturers design guide’ for high efficiency condensing combination space/water heating systems, prepared by the CanmetENERGY Buildings Group, which will help to ensure advanced systems marketed in Canada are appropriate to Canadian conditions, efficient and durable.

#8. Title: Advanced Thermal Heat Pumping Technologies (HPT) Based On Ejector Systems For Houses, Buildings And Communities.

Total Project Value (all sources): $1,070,000 Timeline: 2008-09 to 2011-12

Objective: To develop knowledge and tools to design high efficiency ejector heat pump cycles for heating and cooling applications in the built environment.

Target Audience: Scientists and academics; consulting engineers; heating, cooling, air conditioning and refrigeration systems manufacturers, distributors and users; and energy service companies.

Federal Lead: NRCan, CanmetENERGY – Varennes

Partners: VÉOLIA Environnement; CARNOT réfrigération; RACKAM; LTE-Hydro-Québec; Sherbrooke University.

Summary: An ejector-based heat pump system can be driven by low temperature energy (such as waste heat or heat from renewable resources), produce heat up-grading, cooling and refrigeration effects and has several advantages over conventional heat pump systems including the possibility to use non-conventional refrigerants (such as CO2), simplicity (i.e., no moving parts), ease of operation, and low capital and maintenance costs. However, the use of ejectors for heat pumps has been limited due to insufficient knowledge regarding their performance over a wide range of operating conditions and because of their low energy efficiency compared to conventional vapour compression systems.

Key project activities were the experimental evaluation of one- and two-phase ejector technology performance and further development and validation of tools to design advanced thermal HPT based on ejectors systems. Research on the one-phase ejector system progressed more than originally anticipated due to interest from industry and the establishment of a partnership to develop a large scale prototype and conduct field tests. Focus on the one-phase ejector contributed to modifying the original research plans for the two-phase ejector and not all deliverables were fully realized.

The research resulted in improved design of conventional ejectors and showed that fluid selection and cycle design can improve performance, which indicates their potential for use in heating and cooling systems. Experimental results were used to validate detailed numerical models using computational fluid dynamics and other methods. Overall, progress was made in terms of moving from more exploratory research to the development of large scale prototypes. Researchers have reinforced their expertise in/and partners their understanding of/ ejector technology and their applications. Promising test results have led to increased stakeholder interest and partnerships involving ejector technology; a Canadian manufacturer is field testing the prototype ejector cooling system.

Some lessons learned/best practices:

  • Tripartite collaboration (university/Industry/NRCan) is very important and bridges the gaps between fundamental research and application.
  • Summaries of research activities tailored to target audiences facilitate engagement.
  • Provide flexibility for timelines in R&D - each project is different and the nature and timelines for progress cannot be predicted

#9. Title: Local Energy Efficiency Partnerships (LEEP) And Technology Adoption Pilot (TAP) Demonstration Project.

Total Project Value (all sources): $12,923,044 (Includes $11,800,000 in-kind contributions for the building of demonstration homes) Timeline: 2009-10 to 2011-12

Objectives: 1) To formalize the LEEP process and engage local builders so they can identify and learn about the energy saving and renewable technologies of most interest to them; and 2) to create a field trial process (TAP) that seeks to connect the technology identification process to marketplace adoption by demonstrating LEEP technologies in a series of demonstration houses.

Target Audience: Primarily builders. Secondly municipal building officials; CanmetENERGY researchers; manufacturers; regional home builder associations (HBAs); and utilities.

Federal Lead: NRCan, CanmetENERGY Ottawa Research Centre

Partners: EnerQuality (delivery agent); Four Ontario regional Home Builders’ Associations (Vaughan, London, Sudbury and Hamilton-Niagara); Ontario Power Authority (OPA); Enbridge Gas Distribution; Union Gas.

Summary: LEEP/TAP is a builder-led technology assessment, selection and field trial process. LEEP is delivered through a series of facilitated workshops by which builders use comparative technical assessment resources provided by CanmetENERGY to evaluate and select innovative technologies of interest to them. Builders could then trial LEEP technologies in newly built houses in the TAP phase of the project, during which they received technical support from building scientists, manufacturers and trades people.

Key project outputs were: a developed process and delivery of LEEP/TAP in 4 Ontario municipalities; facilitation training materials; technology assessment resources including modelling and simulations; 14 presentations by manufacturers/others on selected technologies for field-trial; the construction of 40 demonstration homes; 75 case studies; and an “Innovation Forum”.

Key outcomes of the project include an effective approach to increase awareness and understanding of energy efficient practices and technologies among builders and other stakeholders; new and strengthened relationships among stakeholders in the housing industry; increased capacity among builders and manufacturers to integrate new technologies into homes; and a cost reduction of a trialed technology (zoning). LEEP/TAP was considered a valuable step on the path to get new technologies adopted in the marketplace. Several stakeholders are using project results – e.g., CanmetENERGY has applied the project concept of “Minimum Viable Research” to other initiatives; one builder has installed TAP-trialed zoned air handlers into a 117-home development and another is offering TAP-trialed technologies as an option in their new home packages. Finally experience with the process reaffirmed one utilities’ decision that their new energy conservation and demand management program should be builder driven, rather than prescriptive.

Some lessons learned/best practices:

  • Delivering the workshops consecutively in the four regions allowed learning to be passed on.
  • Providing guidelines to presenters helps ensure they are targeting builders’ information needs.
  • The LEEP/TAP process demanded a large time commitment from participant builders; now that many technology assessments have been completed it will be possible to reduce the number of workshops. This would also reduce costs/session and allow new sessions to expand reach.
  • Be clear on the roles and responsibilities of all parties involved in the delivery of the project to avoid inefficiencies

#10. Title: Northern Sustainable Housing

Total Project Value (all sources): $768,000 (plus in-kind and some financial contributions from housing corporations and territorial governments) Timeline: 2008-09 to 2011-12

Objective: To provide support to northern housing providers (housing corporations, Territorial governments and builders) in the design, planning, construction, monitoring and reporting of four energy efficient, healthy, culturally appropriate and affordable homes in the Yukon, Northwest Territories and Nunavut.

Target Audience: Territorial housing corporations; local builders.

Federal Lead: Canada Mortgage and Housing Corporation (CMHC)

Partners: NWT Housing Corporation; Tr’ondek Hwech’in; Arctic Energy Alliance Group; Yukon Housing Corporation; Nunavut Corporation

Summary: The growing need for housing in the North (particularly given increased sovereignty and resource extractions interest in the region) combined with high construction and operating costs is pressuring Northern housing providers to consider different approaches to the design and construction of Northern housing. This project provided support to northern housing providers to develop culturally appropriate, energy efficient housing designs. Project outputs included: design charrettes; housing designs and architectural drawings (using the integrated design process and charrettes); mechanical systems designs; energy modeling and monitoring; reporting. Two E2 and one E9 demonstration homes were built (E/2 housing consumes 50% less energy than similar housing constructed to the Model National Energy Code for Houses (MNECH), and E/9 houses consume 90% less.). One planned demonstration home was not completed within the time frame of the project due to issues with the contractors.

Performance monitoring showed that the E2 houses could achieve the 50% of MNECH target or better. Energy simulations of the E9 home revealed that achieving a 90% reduction in energy use relative to MNECH could not be achieved without implementing sophisticated and costly renewable energy strategies, and that a 70% reduction was more in line with the budget of the First Nations housing provider. The key project outcomes were: demonstrated opportunities to engage northern stakeholders to improve residential energy efficiency; increased awareness and understanding of northern sustainable housing challenges and solutions among target groups; enhanced collaboration among key stakeholders; and new capacity of northerners to build energy efficient houses. There is evidence that project knowledge has influenced technical and design decisions by at least three partners in subsequent northern housing projects and has influenced NWT housing policy and energy efficiency targets.

Some lessons learned/best practices:

  • Consider local economic benefits of any housing project – e.g., using pre-fabricated building components may benefit the project by reducing on-site construction costs and improving energy efficiency and building performance, but provides more limited economic benefit to the local community; this may create tension between energy efficiency and local/regional economic development goals.
  • Innovation in the North should include implementing existing, proven, technologies in novel, northern housing designs, and the development of new processes / approaches to housing design and construction rather than just the application of technologies and systems.

#11. Title: Solid State Lighting (SSL) in The Office of The Future

Total Project Value (all sources): $1,302,000 Timeline: 2008-09 to 2011-12

Objective: To identify and develop novel lighting applications and undertake related human factors research (i.e., research on users’ response to LED light effects), to facilitate long-term market adoption in the office lighting market.

Target Audience: Industry; university researchers; international research organizations; and national and international standards-setting groups.

Federal Lead: National Research Council (NRC) - Institute for Research in Construction (IRC)

Partners: NRCan-Canmet Energy Technology Centre (CETC); BC Hydro; over 20 light source and luminaire manufacturers and lighting designers; Carleton University, UBC, one UK and one US university.

Summary: Currently, solid state lighting (SSL) systems are more energy efficient than incandescent, halogen and linear fluorescent lighting. Research has shown that additional SSL efficiency gains are possible and that by 2020 LED lighting systems will use at least 50% less energy than typical fluorescent fixture. However any operational (energy) savings are currently outweighed by the up-front cost differential between LEDs and standard light sources. This project brought together a consortium of stakeholders to identify possible novel applications of SSL technology in the office environment and address related research needs. Project activities involved i) experimental research on end-users’ interaction with SSL systems; and assessed preferences for light levels, colour quality, colour spectrum, and the effects of flicker on users’ behaviour and cognitive performance; and ii) two full-scale demonstrations showing a variety of possible control options for a corridor and an artificial sky demonstration; and a portable, 1/10th scale model to demonstrate SSL technology at a number of events.

The project succeeded in bringing together the wide range of technical and scientific expertise needed to meet research objectives, which was an important early project outcome. New research partnerships were also established between with NRC and UBC on colour quality research, and Carleton University, Northeastern University and University of Essex on flicker research. The experimental research helped researchers develop a better understanding of the impact of LED lighting on office workers’ productivity, health and general well-being. Demonstration activities and research results have led to an increased acceptance by stakeholders of the importance of human factors and the integration of intelligent lighting systems / controls to the future of the industry. Findings have contributed directly to the work of 4 international standard committees where recommended practices and standards related to SSL are developed.

Some lessons learned/best practices:

  • The SSL project built on lessons-learned during the introduction of CFLs when human factors-related issues presented major barriers to consumers’ acceptance and slowed market development.
  • Convening a group of industry and other stakeholders at the beginning of the project to help identify possible areas of research and develop options for future lighting systems contributed to a high-level of engagement throughout the project.

#12. Title: Advanced Durable Wall Systems: Identification of Wall Frame Assemblies to Meet Net-Zero Energy Housing Targets

Total Project Value (all sources): $427,000 Timeline: 2008-09 to 2011-12

Objective: To improve the overall thermal performance of wall systems by designing and field-testing a novel building envelope system prototype (Vacuum Insulation Panels –VIPS) and preparing guidance material on existing advanced wall technologies and construction techniques.

Target Audience: Architects; builders and manufacturers of advanced wall systems or components.

Federal Lead: NRCan – CanmetENERGY – Housing, Buildings and Communities

Partners: NRC-Institute for Research in Construction; Canada Mortgage and Housing Corporation (CMHC); Two private sector companies; BC Hydro.

Summary: Most current housing and building activity in Canada still uses Code minimum envelope systems despite the demonstrated technical feasibility of more highly-performing, energy efficient building envelopes. Barriers associated with high-R wall systems (R-27 to R-35) include cost premiums associated with additional labour and materials, loss of liveable floor space, and lack of builder awareness and training. New technologies such as VIPs have high thermal resistance: a 1” thick VIP wall system has the same R-value as a 16” conventional fibreglass insulated wall. VIPs, however, are relatively expensive, their long-term performance is unknown, they are very fragile which presents challenges to construction-site installation (a small pinhole will eliminate the thermal insulating capacity of VIPs).

This project carried out applied R&D, demonstration and monitoring of VIPs, and developed construction guidelines with specifications for innovative wall assemblies. Key project outputs were a literature review of insulation materials and typical wall system applications; assessment of the effect of thicker, high-insulation walls on the available living space in homes; documented design, construction and field performance of a full-scale wall assembly incorporating VIPs; construction plans and monitoring results from the demonstration installation; designs and construction strategies for Canadian conditions and climate; and a best practices guide and construction modules for 28 high-R wall systems (delivery was delayed until 2013). Project outcomes include: i) Increased awareness and understanding among researchers, partners, the general public and housing professionals about the possibilities and barriers of VIPs in wall assemblies; ii) Enhanced collaboration among federal stakeholders, sub-program researchers, and between NRCan and private sector companies; and iii) project results are contributing to the development of the Canadian Construction Material Centre test guide for VIPs as a first step in developing a certification scheme for VIPs, and are being used as the basis for two post-secondary architectural courses.

Some lessons learned/best practices:

  • Technology demos require approvals so include building inspectors and city building officials in project design, delivery and outreach activities
  • High-end niche markets (e.g., heritage renovations) are a good way to introduce new materials and technologies while they are still expensive and in the early market development stage
  • The structure of the \Building Envelope Sub-program (interdependent projects) supported effective collaboration and cooperation among the federal partners involved in advanced wall systems RD&D