Evaluation of the Energy and Climate Change Policy (ECCP) Program

Audit and Evaluation Branch
Natural Resources Canada
December 18, 2020

Table of Contents

• List of acronyms
• Executive Summary
• Introduction
• Evaluation objectives and methods
• Evaluation limitations and mitigation strategies
• What we found: program design
• What we found: relevance
• What we found: program delivery
• Conclusions
• Appendix 1: Theory of change for the ECCP Program (Policy is not linear)
• Appendix 2: Survey methodology and limitations
• Appendix 3: List of key documents referenced in report
• Appendix 4: Evaluation team

List of acronyms

ADM

Assistant Deputy Minister

AEB

Audit and Evaluation Branch

CCEI

Canadian Centre for Energy Information

ECCC

Environment and Climate Change Canada

ECCP

Energy and Climate Change Policy

ES

Energy Sector

EPB

Energy Policy Branch

EMMC

Energy and Mines Ministers’ Conference

GHG

Greenhouse Gas

LCES

Low Carbon Energy Sector

NRCan

Natural Resources Canada

PCF

Pan-Canadian Framework (on Clean Growth and Climate Change)

SPPIO

Strategic Petroleum Policy and Investment Office

TB

Treasury Board (of Canada)

UNFCCC

United Nations Framework Convention on Climate Change

Executive Summary

About the Evaluation

This report presents the findings, conclusions, and recommendations from the evaluation of the Energy and Climate Change Policy (ECCP) Program. This Program provides policy advice to the Minister of Natural Resources, senior government officials, and other stakeholders to inform decision-making on matters related to energy and climate, including the future pathway for Canada’s energy sector. The evaluation covers the period from 2014-15 to 2018-19. Total program expenditures over this period were approximately $42 million.

Though the Energy Policy Branch has provided this policy function for a number of years, Energy and Climate Change Policy was not designated as a formal Program until the development of Natural Resources Canada’s (NRCan)’s new Departmental Results Framework. Since this time, the ECCP Program has continued to evolve. Given this, rather than focus on achievement of results, the objective of this formative evaluation was to examine the design and delivery of the ECCP Program.

The evaluation of policy functions is relatively uncommon and not well understood. The policy process is inherently complex and non-linear, making it hard to measure and evaluate. In 2018, NRCan’s Audit and Evaluation Branch worked in close collaboration with the Energy Policy Branch to develop an innovative, forward-looking evaluation approach.

The evaluation was completed in two steps:

1. Confirm or revise the ECCP Program’s theory of change, i.e., its results logic, reach and the causal relationships between its activities, outputs and outcomes.
2. Examine the recent design and delivery of the ECCP Program to assess the extent to which this enables it to respond to identified needs and achieve (or contribute to) its intended outcomes. This includes consideration of the internal and external factors that might hinder or facilitate the Program and (to the extent possible) the nature and magnitude of their impact.

The starting frame for this evaluation was the Program description and results logic included in the ECCP Program’s Performance Information Profile (2017) (PIP). Facilitated by an external resource and insights introduced through the conduct of this evaluation, this framework was updated by ECCP Program representatives while the evaluation was in progress.

The results of this evaluation will help inform the design of the ECCP Program as this framework is further updated by the Low Carbon Energy Sector (LCES). The evaluation also provides insights that can be applied to improve the design and delivery of other policy functions at NRCan.

What the Evaluation Found

Program Design

We found that the Energy Policy Branch’s understanding and articulation of the ECCP Program’s design continues to evolve and improve. Measuring the performance of policy functions is a complex activity. Over most of evaluation period, the ECCP Program had a weak performance measurement framework resulting in important gaps in the availability and utility of its performance information. However, since 2018, the ECCP Program has been undertaking substantive efforts to improve its PIP, including the Program description, results logic and performance metrics.

Analysis completed in support of this evaluation confirms the revised results logic for the ECCP Program, i.e., that the Program’s activities, outputs and outcomes have been appropriately identified and that it is reasonable to expect that the Program’s activities should lead to its intended outcomes. Our analysis also expands on progress to date to further the development of the theory of change for the ECCP Program and identify additional options that the ECCP Program could explore as it develops its performance metrics. The measures that will be implemented need to reinforce the value of the Program based on, for example, the nature of stakeholder relationships or its ability to reframe understanding of a policy issue.

Relevance

The ECCP Program is relevant to federal government and NRCan priorities for energy and climate change. Revisions to the Program’s logic model effected in response to the reorganization of the Energy Sector now focus outcomes more clearly on advancing Canada’s low-carbon energy and climate change priorities. There is still a need for further clarification in some areas, such as the extent to which Program priorities should include climate change adaptation.

The evaluation also found that the ECCP Program’s activities are relevant to the needs of key stakeholders. Particularly given shared jurisdiction for energy and climate change, stakeholder engagement is fundamental to the effectiveness of the ECCP Program. Overall, we found that appropriate target audiences have been engaged or reached. It is not possible in the absence of a clear engagement plan to fully assess the adequacy of these collaborations. Regardless, evidence suggests a need to increase engagement in some areas, particularly with those groups that are generally underrepresented in policy development and implementation (i.e., women, youth, and Indigenous peoples). The ECCP Program’s Equal by 30 Campaign is noted as a positive example in this regard.

Program Delivery

This evaluation reviewed the extent to which the conditions and external factors identified in the theory of change are being met by the ECCP Program and/or are otherwise impacting the delivery of its program activities. Overall, there is evidence of effective program delivery but with some room for improvement. Across the five categories included in the theory of change, we found that:

• Process Management: There is evidence of effective process management in response to stretched resources, with human and financial resources allocated as required to address priorities. However, these resources may not be aligned with current or future work demands given the increasing number and scope of federal energy and climate change initiatives. Evidence suggests that the Energy Policy Branch could be more effective at securing durable funding to support these initiatives. While the competency of existing staff is considered a strength of the Program, there have been issues with recruitment and retention of staff including for senior management (EX) positions. The reasons for this are not entirely clear.
• Quality of Policy Products: There is evidence that the ECCP Program produces quality policy products that are neutral and evidence-based. This is critical to meaningful dialogue on energy and climate change. There may be some opportunities for the Program to better reflect the perspectives of other NRCan sectors in its internal products. However, the subject matter is complex and requires a balance of interests to ensure policy products reflect those sometime divergent perspectives. There is also evidence that the ECCP Program produces quality policy products and that these are reaching target audiences. External factors (e.g., political relationships, stakeholder capacity, etc.) impact the extent to which these products influence stakeholder perspectives or result in concrete actions. The ECCP Program is actively working to launch the Canadian Centre on Energy Information, which is expected to address current gaps in access to reliable, transparent and science-based information as the evidence base to make informed decisions. This should enable both Canadians and the ECCP Program to conduct stronger economic and energy analysis.
• Stakeholder Engagement: Canada’s low-carbon energy transition cannot be achieved without collaboration. There is evidence that the ECCP Program’s policy products are reaching target audiences, but external factors (e.g., political relationships) impact the extent to which these products influence stakeholder perspectives or result in concrete actions. The ECCP Program also works to engage stakeholders and be inclusive of stakeholder perspectives in key policy exercises but some stakeholders (particularly underrepresented groups) may lack the capacity or adequate opportunities to participate. There is thus a risk that the perspectives of some stakeholders are not being collected and conveyed or that opportunities to collect fresh perspectives or new solutions are being missed. As illustrated by Generation Energy, we also found that engagement is not an end in itself. While the pathways indicated by Generation Energy have become a foundation for NRCan’s advice to the government on how to advance Canada’s low-carbon energy transition, changes to the broader policy environment impacted NRCan’s ability to begin implementing its specific recommendations. Stakeholders need to see concrete actions or there is a risk that they may disengage.
• Policy Environment: The extent to which the policy environment is supportive of the ECCP Program and its outcomes is a key external factor impacting on program delivery. As with all policy programs, policy development within the ECCP Program is largely influenced by government priorities, which can change or evolve quickly depending on the environment.
• Relative Advantage: As per the ECCP Program’s theory of change, its ability to contribute to advancing Canada’s low-carbon energy and climate change priorities requires internal and external stakeholders use its advice and analysis in decision-making. Their motivation or desire to do so also assumes that there is a real or perceived advantage to be gained from adopting the resulting policy, decision or practice. The ECCP Program currently plays an important role in understanding and integrating stakeholders’ polarized views to develop responsive policy options or solutions. Moving forward, its policy advice must continue to recognize the inherent tension between priorities for energy and the environment and actively identify opportunities to bring the two together.

Recommendations and management response

Recommendation	Management Response
The ADM of LCES should complete its update of the ECCP Program’s PIP, including revision of the program theory and results logic. The ADM of LCES should also support the ECCP Program in developing and testing indicators to facilitate their use as meaningful metrics of program performance.	Agreed. The ECCP Program’s PIP will be updated. This update will include a revised logic model, theory of change, and at least one indicator that is more relevant to measuring meaningful program performance. As the indicator is more experimental than in previous years, results will be tested and possibly adjusted over time based on feedback.  Due date: December 31, 2020.
Working as required with other sectors, the ADM of LCES should build on the learning from this evaluation to develop and maintain a stakeholder engagement plan for the ECCP Program. This plan should consider the following elements: Who needs to be consulted / engaged, and for what reasons; The nature or level of collaboration desired in engagements; Mechanisms to engage underrepresented groups; Mechanisms to maintain networks of key contacts; and Feedback loops required to inform stakeholders of the results of collaborations.	Agreed. In 2020-21, LCES developed stakeholder engagement plans for individual policy priorities, including: the development of a hydrogen strategy and of a Small Modular Reactor action plan; consultations on radioactive waste management. Starting in 2021-22, LCES will include, as part of its annual business plan, a stakeholder engagement plan which will reflect the elements identified in the recommendation. This overall stakeholder engagement plan will provide an integrated approach to managing these activities within the context of the broader priorities of the sector. Due date: August 31, 2021.
The ADM of LCES should conduct an internal review of the ECCP Program’s business processes to identify resources and tools available for program delivery, and develop a plan to ensure that its capacity to deliver continues to be aligned with activities.	Agreed. Following the departmental reorganization that created LCES in 2019, the new Sector undertook a review of its operations which led to a reorganisation of the Branch, which was announced in June 2020. The reorganisation included the integration of the Energy Policy Branch with the international functions of the International Energy Branch, in recognition of the evolving and transforming policy space in which the Program works. A review of the operations of the new Energy Policy and International Affairs Branch is currently underway to ensure resources and tools are well aligned across its various work areas. This review will result in a plan to ensure branch capacity is aligned with its activities, including in light of its role to coordinate the implementation of NRCan’s measures in support of the government’s Strengthened Climate Plan. Due date: June 30, 2021.
The ADM of LCES should “close the loop” on Generation Energy by providing stakeholders that engaged in the dialogues periodic updates on the actions taken to date and/or future government commitments.	Agreed. The Program plays a key role in advancing the government’s priorities related to the transition to a low-carbon economy and addressing climate change, which were informed by the Generation Energy process and recommendations. The Program will develop proposed approaches to communicate to stakeholders and the public how Generation Energy informed various elements of the government’s Strengthened Climate Plan. Due date: June 30, 2021.

Introduction

This report presents the findings, conclusions, and recommendations from the formative evaluation of the Energy and Climate Change Policy (ECCP) Program. The evaluation examines the delivery of the program over the period from 2014-15 to 2018-19. The Audit and Evaluation Branch (AEB) of Natural Resources Canada (NRCan) undertook the evaluation between December 2018 and February 2020. It was conducted in accordance with the Treasury Board (TB) Policy on Results (2016).

Program information

Though the Energy Policy Branch has provided this policy function for a number of years, it was not designated as a formal Program until the development of NRCan’s new Departmental Results Framework. The ECCP Program is now Program 2.11 in NRCan’s Program Inventory. This Program provides policy advice to the Minister of Natural Resources, senior government officials, and other stakeholders to inform decision-making on matters related to energy and climate, including the future pathway for Canada’s energy sector. It serves as the federal government’s central energy policy hub in the transition to a low-carbon, sustainable, inclusive and competitive resource economy.

Within its four core functions, the ECCP Program:

1. Conducts economic research and analysis to support policy-making and provide the public with data, forecasting, and other information on Canada’s energy sector. This activity produces tools and information products for decision-makers and the public, such as NRCan’s Energy Fact Book and the recently announced Canadian Centre for Energy Information (CCEI).
2. Conducts and consolidates policy research and analysis across energy sub-sectors (e.g., oil and gas, energy efficiency, renewables, nuclear, technology and innovation, etc.) on both energy and climate change-related issues. Key outputs of this activity include guidance, advice, and information products (e.g., studies, briefings, and policy papers) for internal decision-makers and the products of policy development (e.g., Memoranda to Cabinet and TB Submissions).
3. Coordinates with department officials in both NRCan and other government department on NRCan’s participation in government-wide energy and climate change initiatives such as the Canadian Energy Strategy and Pan-Canadian Framework on Clean Growth and Climate Change.
4. Collaborates with energy stakeholders. Central to the ECCP Program, this includes engagement with relevant stakeholders to inform government priorities, develop and influence government agendas, promote knowledge exchange through networks and partnerships aimed at reducing greenhouse gas emissions, and to develop, influence and promote policy options that drive the sector towards a competitive and low-carbon energy future. Through these engagements, the Program works to collect and convey stakeholder perspectives to decision makers. These collaborations can be either ongoing (e.g., the annual Energy and Mines Ministers’ Conference [EMMC]) or ad hoc (e.g., Generation Energy dialogues).

Work to revise the presentation of the Program’s logic, including articulation of its expected outcomes, is still in progress (see section on Program Design).

Program Governance

This Program is under the responsibility of the Director General of the Energy Policy Branch. This Branch consists of three Divisions: (1) Strategic Energy Policy; (2) Energy and Environment Policy; and (3) Energy and Economic Analysis. Each division provides specialized policy services and expertise that support the energy transformation agenda (Figure 1).

Figure 1: Role of Energy Policy Branch Divisions

For most of the evaluation period, the Energy Policy Branch reported to the Assistant Deputy Minister (ADM) of NRCan’s Energy Sector. In October 2019, a departmental reorganization divided responsibility for tackling priorities related to energy into three teams:

• Low Carbon Energy Sector (LCES), leading on strategic energy policy, international energy files, electrification, renewable energy and energy efficiency. 
• Strategic Petroleum Policy and Investment Office (SPPIO), with a mandate to bring greater focus to the strategic priorities of Canada’s oil and gas sector in both international and domestic markets. The Office also includes the CanmetENERGY Research Centre (Devon, Alberta), working to strengthen energy partnerships with industry, provinces and academic institutions to better integrate the research driving science and innovation in the oil and gas sector, and a new special advisory role to manage emerging energy issues. The ADM SPPIO also continues to provide leadership on files related to specific major electrification projects, such as Lower Churchill and BC electrification initiatives.
• Energy Technology Sector (ETS), leading research and development to provide clean energy solutions connected to energy policy and innovation.

This reorganization recognizes that access to reliable, affordable and sustainable energy is among the top priorities for Canadians and investors. The work of these three teams is to be coordinated through a new Energy Future Committee chaired by the Deputy Minister to deliver results for Canadians.

The Energy Policy Branch is now under the responsibility of the ADM LCES. At least during this transition period, it continues to also provide policy services as required to support the ADM SPPIO. While the Branch works in close collaboration with the Energy Technology Sector, it does not provide direct policy support to this sector.

This Branch also works alongside the policy support provided elsewhere both within LCES, such as that of the International Energy Branch and of the Office of Energy Efficiency’s Demand Policy and Analysis Division, and across NRCan (e.g., ETS’ Office of Energy Research and Development; Lands and Minerals Sector’s Climate Change Impacts and Adaptation Division; and Canadian Forest Service’s Science Policy Integration Branch).

Evaluation objectives and methods

This evaluation examines program delivery over the period from 2014-15 to 2018-19 (with consideration for more recent events). Since its formal designation in 2017, the ECCP Program has continued to evolve. Given this, rather than focus on achievement of results, the objective of this formative evaluation was to examine the design and delivery / implementation of the Program, and related implications for its capacity to respond to identified needs and achieve intended outcomes.

Specific evaluation questions were as follows:

Program Design

1. What is the results logic of the ECCP Program?
2. What are the causal relationships between the ECCP Program’s activities, outputs and its intended outcomes?
3. What factors impact the ECCP Program’s ability to respond to its intended needs and achieve intended outcomes?
4. To what extent is the ECCP Program collecting (or able to collect) performance information that is (or can be) used to assess its effectiveness and efficiency?

Relevance

5. To what extent does the design and delivery of the ECCP Program respond to, or align with, federal government and departmental priorities?
6. To what extent does the design and delivery of the ECCP Program respond to, or align with, the needs of key stakeholders?

Program Delivery

7. To what extent is the ECCP Program delivered in a way that will facilitate the achievement of outcomes?
8. To what extent are the human and financial resources of the ECCP Program aligned to support the delivery of relevant federal government and NRCan priorities?

In 2018, AEB worked in close collaboration with the Energy Policy Branch to develop an innovative evaluation approach that would be formative in nature (i.e., forward-looking).

The evaluation was completed in two steps:

1. Identify or confirm the Program’s key stakeholders and their needs for the Program (including relevant government and departmental priorities), and confirm or revise the Program’s theory of change, i.e., its results logic and the causal relationships between its activities, outputs and outcomes.
2. Examine the design and delivery of the Program since 2014-15 to assess the extent to which this enables it to respond to identified needs and achieve (or contribute to) its intended outcomes. This includes consideration of the causal relationships and different stakeholders involved or impacted at each level of the logic model and of the internal and external factors that might hinder or facilitate the Program and (to the extent possible) the nature and magnitude of their impact.

The evaluation of policy functions is relatively uncommon and not well understood. The design of this evaluation draws from the methodological practices outlined in the TB Secretariat’s draft guide on Evaluating Policy Functions: Exploring Concepts and Tools. The validity of the evaluation approach was also confirmed by an external advisor contracted for his expertise on evaluating policy programs. The evaluation further leveraged the knowledge and experience of this external resource to facilitate the integration of related best practices.

The Energy Policy Branch has remained actively involved during the conduct of this evaluation, and has worked closely with the same external resource to develop a new theory of change for its policy function, including the introduction of new approaches to measure the effectiveness of policy work. Results of this evaluation will help inform the design of the ECCP Program as it is updated by LCES.

Formative evaluation lends itself to qualitative methods of inquiry. This evaluation used four key lines of evidence, consisting primarily of qualitative data (Table 1).

Table 1: Evaluation Methods

Evaluation limitations and mitigation strategies

• The starting frame for this evaluation was the Program description and results logic included in the ECCP Program’s Performance Information Profile (PIP). Recognized as deficient, this framework was updated by ECCP Program representatives while the evaluation was in progress. These updates were facilitated by an external resource and learnings introduced through the conduct of this evaluation. Related revisions, and additional changes to the Program’s design that occurred during the conduct of the evaluation (e.g., transfer of the ECCP Program’s international activities and outcomes to the International Energy Engagement Program, changes in senior leadership at NRCan impacting the ECCP Program’s governance structure) are considered in the evaluation’s analysis.
• A detailed theory of change for the ECCP Program was developed as part of this formative evaluation. As a result, there was limited information related to the extent to which the ECCP Program aligns to some conditions and external factors, as these had not yet been identified at the outset of the evaluation.
• While the evaluation conducted extensive interviews, the ECCP Program has a very broad range of targeted stakeholders. In particular, the views of representatives from industry or environmental non-government organizations may be underrepresented in the interview analysis. Where available, we mitigated this limitation by considering secondary evidence of stakeholder needs and perspectives found in program documents and literature. 
• Many were generally familiar with the Energy Policy Branch, but the ECCP Program itself is not well known or understood by internal or external stakeholders. This limited the ability of these stakeholders to provide informed perspectives on program design. To mitigate this limitation, evaluators provided all interview respondents with a placemat summarizing these key design elements and results logic based on the information in the most current PIP. To the extent possible, stakeholder perspectives on program delivery were also validated against evidence presented in the document review.
• Specific challenges were encountered in the conduct of the evaluation’s surveys. These challenges and resulting limitations are outlined in Appendix 2.

What we found: program design

As identified in the Treasury Board of Canada Secretariat’s draft document on Evaluating Policy Functions: Exploring Concepts and Tool, building an understanding of policy processes is an important first step in planning evaluations of a policy function such as the ECCP Program (see Box 1).

Box 1: Defining the “Policy Function”

The starting frame for this evaluation was the Program description and results logic included in the ECCP Program’s PIP (2017). Facilitated by an external resource and insights introduced through the conduct of this evaluation, this framework was updated by ECCP Program representatives while the evaluation was in progress. Figure 2 provides a visual presentation of the Program’s current results logic.

Analysis completed in support of this evaluation confirms the revised results logic for the ECCP Program, i.e., that the Program’s activities, outputs and outcomes have been appropriately identified and that it is reasonable to expect that the Program’s activities should lead to its intended outcomes. However, the results pathway for policy functions is complex. While presented as a linear model, the Program’s activities are actually iterative and inter-related, not discreet or linear. For example, stakeholder perspectives collected as part of engagements are used to improve information products, which in turn will be used to frame policy and refine the problem being addressed. As a result, multiple results chains with dynamic interactions are likely realized. These chains also interact with external factors that may influence the expected results. This is consistent with literature-based theory on policy development.

Figure 2: Logic Model for the Energy and Climate Change Policy Program

Text version

Figure 2. Logic Model for the Energy and Climate Change Policy Program. The logic model is shown as a series of boxes in five columns proceeding from left to right. The first column, activities. One, conduct economic and data analysis. Two, conduct policy research and analysis. Three, coordinate with departmental officials. Box four, collaborate with energy stakeholders. Proceeding to the second column, outputs. One, tools and information products for decision makers and the public. Two, stakeholder perspectives. Proceeding to column three, immediate outcomes. One, federal government decision-makers and stakeholders have access to tools and information to support decisions and negotiations on key energy issues. Two, engagement and awareness of appropriate and necessary federal government stakeholders. Three, engagement and awareness of appropriate and necessary domestic stakeholders. Proceeding to column four, intermediate outcomes. One, use of Energy and Climate Change Policy Program advice and analysis in decisions and policies by Canadian federal government decision and policy makers. Two, use of information in decisions, policies, and practices by domestic stakeholder groups. Proceeding to the final column five with a single box, ultimate outcomes, Canada’s low-carbon energy and climate change priorities are advanced. One rectangular element labelled assumptions and factors, underlies all phases of the logic model with two-way connections between the assumptions and factors and the rest of the logic model.

As shown in Figure 2, recent updates to the Program’s PIP effected by the Energy Policy Branch also recognize that this policy function does more than just produce information for decisions makers. The theory of change for the ECCP Program is now described as three streams of activities (or distinct functions), each providing its own value proposition. These include:

• Policy as a professional service: This area relates to the activities and outputs that are the core work of any policy group. This includes organizing data, conducting research and analysis, and preparing written and oral presentations to ensure internal and external stakeholders have the information they need to make decisions. For the ECCP Program, it also includes providing policy support to senior management and central agencies to advance key energy priorities, coordinating and reporting on NRCan’s contributions to Government-wide energy and climate change initiatives, and conducting economic and statistical analysis of energy issues for departmental officials, stakeholders, and international partners. This category of work supports the higher-level outcomes related to policy as a convening function and policy influence and oversight.
• Policy as a convening function: This area relates to the ECCP Program’s activities, outputs and immediate outcomes. It includes ensuring resources are appropriately applied to consulting, coordinating and other key functions to enable a collaborative forum or group that represents a target community on a specific energy- or climate change-related issue. This engagement approach enables two-way constructive feedback of participants to become aware of the policy development process and to provide input into related proposals. Convening is now recognized as critical to the results of the ECCP Program. This is a fundamental shift in program design; pre-existing logic models lacked any trace of effective engagement as part of the program theory.
• Policy influence and oversight: This area relates to the ECCP Program’s activities, outputs, immediate and intermediate outcomes. It includes various activities that lead to key policy outputs (e.g., guidance or options), and which reach a key target community or user. These groups are reached and engaged in a constructive dialogue, which leads to an improved understanding of the policy impact and approach. Through information sharing, evidence and constructive dialogue, those influenced by the approach make decisions to accept and implement the policy, leading to broader systemic changes in the energy system.

The evaluation used evidence collected from each of its methods to further the development of the theory of change for the ECCP Program (Appendix 1). This model considers:

• The ECCP Program’s results logic (logic model);
• The causal relationships that explain the results logic;
• The conditions and external factors that influence its intended outcomes; and
• The ECCP Program’s spheres of influence.

Box 2: What is a Theory of Change?

There are numerous conditions and external factors that influence the ECCP Program’s intended outcomes. The model in Appendix 1 regroups these into five broad categories:

Spheres of influence are also an important consideration in this analysis (Figure 3). All conditions and contextual factors – including those that cannot be controlled or influenced – will impact program effectiveness. While many elements in the theory of change for the ECCP Program are within its direct control or influence, as with all programs, as you move up the results chain this control weakens and it becomes harder to attribute results directly to program interventions. For example, the ECCP Program can control who it invites to participate in an engagement but cannot control which stakeholders decide to participate. Similarly, it can influence the uptake of advice and analysis by promoting awareness of its products or tools, but cannot control the extent to which stakeholders use this information to make decisions. Additional actions the ECCP Program can take to increase its relevance or improve its design and delivery must consider this influence, but need not necessarily be limited to the sphere of “direct control”.

Figure 3: Spheres of Program Influence

We encourage the ECCP Program to consider the conditions and external factors in the theory of change presented in this evaluation report as it moves forward with reviewing and revising its program design and performance metrics.

There are a number of theory-based approaches to evaluation, with one common feature – all develop a theory of change for the intervention and then verify the extent to which the theory matches what is observed. Similarly, the next step in our analysis was to review performance, i.e., the extent to which the conditions identified in the theory of change are being met by the ECCP Program and/or are otherwise impacting its delivery.

Literature supports both the value and challenge of measuring the performance of policy functions. As described by the Office of the Auditor General (OAG) Manitoba in its Guide to Policy Development (2003), in the absence of information on how previous policies or the policy process have worked, policy development can be an exercise of “shooting in the dark” and risks perpetuation of policy approaches that may not be working. Relying on complaints or ad hoc feedback from clients and stakeholders can create a false sense of security.

Yet, as discussed, the policy process is complex. Approaches to performance measurement that work well for the rest of NRCan’s programs may not work as well when applied to a policy program. There is an inherent, acute challenge in attributing outcomes to the specific activities or outputs of a policy function. Metrics tend to be quantitative and focus on production of outputs rather than achievement of outcomes, or focus only on decisions and actions (e.g., proportion of recommendations adopted), which can be heavily impacted by external factors.

Over most of the evaluation period, the ECCP Program has had a weak performance measurement framework. While it was able to produce metrics required for corporate reporting (e.g., NRCan’s contributions to the Federal Sustainable Development Strategy or Pan-Canadian Framework), these publically reported metrics tend to focus on policy implementation (program outcomes), not policy development. Evaluation evidence also suggests that the Program has lacked adequate tools, processes and mechanisms to effectively monitor and report performance. There are thus important gaps in the availability and utility of its existing performance information.

However, since 2018, the ECCP Program has been undertaking initiatives to improve its performance metrics. For example, its recent survey on the knowledge and uptake of the Energy Fact Book (see Appendix 2) was a test of the utility of this survey tool as an indicator of the extent to which recipients (i.e., decision-makers or influencers), directly or indirectly, use policy products, initiatives and/or activities to inform or advance a desired policy outcome. Given the challenges in assessing the performance of a policy function, the ECCP Program needs to be supported and given the space to innovate and experiment with new metrics. Informed by this evaluation, the measures that will be implemented need to reinforce the value of the Program based on, for example, the nature of stakeholder relationships or ability to reframe understanding of a policy issue, and not the number of products (see generic examples in Table 2).

The ECCP Program has also recently added metrics related to the CCEI to its PIP. While the feasibility of data collection based on these metrics has yet to be tested, they do show potential as useful performance metrics related to both the quality and uptake of energy information.

Table 2: A Basic “Menu” of Metrics for Measuring the Results of Policy Programs

Result	Influence of Policy Program on Result	“Menu” of Policy Result Metrics
Engagement	Building networks and partnerships that support the delivery of change. Promoting dialogue exchange and learning among (network) members. Convening organizations or people (who would not otherwise communicate with each other in this way.)	Type of engagement (e.g., network, cooperating, coordinating, collaborating). Extent of engagement (e.g., level of participation, coverage of groups). Quality of engagement (e.g., directness, clarity or timeliness of communication). Degree of relationship building (e.g., openness, information sharing, and transparency among actors).
Reactions	Changing perceptions – increasing awareness and shaping stakeholder opinion. This includes but is not limited to user satisfaction.	Type (positive or negative) and level (extent of positive or negative) of thoughts, feelings and reactions from the experience.
Learning	Agenda setting – changes in policymakers’ priorities, with attention to previously underemphasized issues. Developing capacity – improving high-level understanding of an issue, and improving how policymakers respond. Shifts in policy framing – changes in the way that policymakers understand a problem or the possible responses to it.	Level of changes to awareness, knowledge, understanding, attitudes, skills, aspirations, commitments and /or priorities.
Decisions and Actions	Changing institutions – influencing policy, strategy and resource allocations, including developing legal, regulatory and social frameworks. Changes in the way policy is delivered – substantive change in the way policy is implemented and/or the way policy is delivered to intended recipients. Changes in policy content – substantive changes in the content of policy and/or resources allocated.	Level and/or extent of adoption of new and/or specific policy content and/or changes to policies (content), practices, protocols, processes and/or delivery.

Source: Adapted from S. Montague (2014)

The evaluation’s literature review was also useful in identifying additional options that the ECCP Program could explore as it develops its specific performance metrics. Besides direct measures of the Program’s outputs and outcomes, this list suggests measures of conditions influencing outcome achievement identified in the theory of change (Appendix 1). This includes:

• Product Quality. Linked to ‘reactions’ and ‘learning’ in Table 2, this could include qualitative assessments of policy product attributes against set criteria for product quality on a sample of “key” products, either by self-assessment or with stakeholder feedback.
• Collaboration. Linked to ‘engagement’ in Table 2, this could include a survey of levels of collaboration and/or stakeholder satisfaction with engagement. To be truly meaningful, the results of this assessment would ideally be compared to targets or objectives in a stakeholder engagement plan.
• Awareness and Use. Linked to ‘learning’ and ‘decisions and actions’ in Table 2, this could include measures of the knowledge, use and uptake of “key” products. As noted, the ECCP Program is already considering a related indicator and assessing the various methods via which this information can be collected, e.g., stakeholder surveys, citation analysis, and web analytics. 
• Constructive Dialogue. Linked to the ECCP Program’s ultimate outcome (and the importance of ongoing stakeholder collaboration in achieving this end), indicators in this category would reflect on the Program’s ability to effectively use its evidence base and convening function to frame policy and encourage positive systemic changes. The feasibility of measuring the degree of polarization in the energy and climate change policy discourse was recently demonstrated in a survey completed by Positive Energy, supported with funding from the ECCP
  Program^{Footnote 1}. Data collected to support such measures could also be used to identify core challenges and opportunities in building public confidence in government decisions on Canada’s energy future.

Metrics for outputs or immediate outcomes could be relatively easy to operationalize. However, particularly given the importance of external stakeholders in achieving the Program’s outcomes, as one moves up the results chain, these metrics become increasingly complex (but would also be measured with less frequency). While many of these metrics would involve an incremental cost to the Program, they would help shift the focus from the effectiveness of policy implementation (i.e., program outcomes such as emissions reductions) to policy development and could provide very valuable information for program management.

What we found: Relevance

The federal government and NRCan have clear priorities for energy and climate change. The Government of Canada recognizes that strong action to address climate change is critical and urgent. As a party to the United Nations Framework Convention on Climate Change (UNFCCC), Canada ratified the Paris Agreement in 2016, with a target to reduce Canada’s greenhouse gas (GHG) emissions by 30% below 2005 levels by 2030. In 2019, the federal government further pledged to reduce Canada’s GHG emissions to “net zero” by 2050.

The Government of Canada also recognizes that energy will play an integral role in meeting these commitments, given that energy production and use accounts for over 80% of Canada’s GHG emissions (Figure 4). Canada’s energy transition will require using low-carbon energy to power our homes, workplaces, vehicles, and industries, and using energy more efficiently.

Box 3: Low-Carbon Energy

The sustainable development and integrated management of natural resources, including energy, is central to NRCan’s mandate. As outlined in the Departmental Plan 2019-20, NRCan has a core responsibility to lead the transformation to a low-carbon economy by improving the environmental performance of Canada’s natural resource sectors through innovation and sustainable development and use.

While we found the ECCP Program to be relevant across the entire evaluation period, documents indicate a shift in priorities over time. Since fall 2015, the federal government has maintained its focus on increasing market access for natural resources and creating jobs for the middle class. However, it has increased its focus on a strong economy as being linked to a healthy environment. Mandate letters for the Minister of Natural Resources also indicate a responsibility to engage with Canadians and collaborate with stakeholders (particularly provinces and territories), with considerations for inclusion of underrepresented groups. The federal government has also placed an increased emphasis on transparency, open government, and evidence-based decision making across policy fields, including for energy and climate change.

Figure 4: Where Canada’s greenhouse gas emissions come from

We found that the ECCP Program has responded effectively to these shifts and that its role in addressing related challenges has increased the relevance of the Program. For example, the ECCP Program played a key role in leading NRCan’s contributions to federal actions in the Pan-Canadian Framework on Clean Growth and Climate Change (2016). 

While this provides strong evidence of alignment to federal and NRCan priorities, evidence also indicates that over the period of evaluation it was not always clear exactly which priorities the ECCP Program was looking to advance. The challenge of identifying clear program outcomes is one frequently observed in the design of policy programs, with outcome statements that are somewhat vague in theory supporting the flexibility to be responsive to new or shifting priorities as they emerge. While the focus in the ECCP Program’s outcome statements is now more clearly on advancing Canada’s low-carbon energy and climate change priorities, evidence indicates that the Program’s work in some areas (e.g., analyses of competitiveness and energy supply, ongoing policy support to SPPIO) may go beyond this expected outcome. To some extent, this is a legacy issue from when NRCan still had a single Energy Sector. At the same time, the evaluation recognizes that this is a challenging policy space where there is a need to balance the narrative. Program representatives also note that the sustainable development and competitiveness of energy resources and advancement of climate change objectives are not mutually exclusive, and the ECCP Program aims to bring the two together.

As NRCan’s primary program dedicated to climate change policy, most stakeholders believed that the ECCP Program needs to integrate consideration for the department’s full suite of policies related to the intersection of energy and climate change. Specifically, many noted that the Program’s outcome statements are not specific to climate change mitigation. Long-term trends in energy are influenced primarily by factors such as changes in population, economic activity, energy prices, clean energy technology and energy efficiency. However, research indicates that many aspects of energy demand, supply, and transmission are sensitive to climate variability and will be impacted by various dimensions of climate change, including higher temperatures, changing frequency and intensity of extreme events and changes in water availability (Figure 5). The net annual result of climate change-induced shifts will be country and region specific, and will be influenced by the actions taken to adapt to predicted effects.

Figure 5: Examples of predicted climate change impacts on Canada’s energy sector

While there are discussions on climate change adaptation in many of the ECCP Program’s policy products, we observed opportunities for more in-depth presentation of these connections in program outcomes and implementation. NRCan’s Climate Change Adaptation Program (Program 1.9) also provides policy support in this area. Nevertheless, given differences in the drivers and key actors for climate change mitigation and adaptation, internal stakeholders engaged in adaptation suggested that their input should be sought earlier in related policy development and be given more weight in approval of related content for senior officials.

All of the ECCP Program’s activities were found to be important enablers for the Program and relevant to the needs of key stakeholders. Perceptions of the relative importance of each activity varied by stakeholder group, often in reflection of the bias of a particular stakeholders’ mandate. For example, those who work in the area of energy information and statistics were more likely to see economic research and analysis as a critical activity.

The ECCP Program’s economic research and analysis meets a need for decision-makers to access high-quality information to make, validate and communicate policy-related decisions. The ECCP Program plays an important role as an “honest broker” in the provision of non-biased, evidence-based energy information.

However, research also shows that most Canadians have a poor understanding of energy policy and related issues. Ensuring access to neutral, evidence-based information about energy systems and the potential impacts (positive and negative) of diverse approaches to achieving Canada’s energy future is thus critical to meaningful dialogue. This is a need that could be addressed by the ECCP Program, provided it can find levers through which to more effectively communicate to Canadians the role and importance of energy. At present, evidence indicates that the energy information provided by the ECCP Program and other data providers does not adequately meet the needs of key stakeholders. This latter gap is expected to be addressed by the CCEI (see Box 4).

Similarly, there is a strong internal demand for integrated policy analysis and advice. The need for intra- and inter-departmental policy coordination has increased over the period of evaluation as the federal government has introduced new commitments and initiatives related to energy and climate change.

Box 4: The Canadian Centre for Energy Information (CCEI)

The evaluation found that these professional services are mostly reactive (e.g., in response to specific issues, events or information requests). The Energy Policy Branch has also been vocal about the need for more space and time to conduct policy research. While literature suggests that this is relatively typical of a policy function, recently the Branch has undertaken more anticipatory research and analysis work to support program renewals and meet the needs of internal stakeholders. Senior managers have also begun exploring how the ECCP Program can be supported to further develop its capacity to engage in proactive foresight and pre-emptive policy research and analysis exercises.

By contrast, the ECCP Program’s activities related to collaboration with stakeholders are relatively proactive and policy driving. The debate on energy and climate change is heavily polarized. The ECCP Program plays an important role in understanding and integrating stakeholders’ polarized views to develop informed policy options or solutions. The number and breadth of stakeholders that engaged in the Generation Energy dialogues convened by the ECCP Program in 2017 (an estimated 380,000 Canadians) is a clear indication of stakeholder needs in this area. However, the evaluation’s interviewees indicated that this activity could be better linked with outcomes, i.e., that engagement is not an end in itself. Stakeholders need to see concrete actions as a result of their engagements, both to ensure their priorities are addressed and as a pay-off for the investment of their time. This issue is further discussed under Program Delivery.

Literature-based theory on the design of policy programs strongly supports a need to consistently and actively engage a broad community of stakeholders in policy development.^{Footnote 2} Policy issues tend to have horizontal implications, involving more than one department, level of government and/or other external stakeholders. There is also an assumed or, in some cases, a legislated need to consult Canadians to inform government decisions. The quality of policy advice and decisions depends on the integration of these stakeholders and their perspectives into the policy process.

While important for most programs, we found that stakeholder engagement is a fundamental consideration for energy and climate change policy. The ECCP Program is not the only program within NRCan or the Government of Canada concerned with either issue. Similarly, jurisdiction for energy policy is shared between the federal and provincial governments. Climate change also has the potential to impact everyone. Energy and climate change is thus a crowded policy space with a low barrier to entry (e.g., where a single teenager can start an entire movement), within which stakeholder groups are rapidly evolving.

Developing and maintaining stakeholder relationships takes time and effort. The large number of governance bodies for energy and climate change within the federal government itself, let alone external networks, makes this engagement environment very complex and energy intensive and, even where there is concerted effort to do so (e.g., the Equal by 30 Campaign), the ECCP Program is challenged to maintain its network of external stakeholder contacts. Numerous external stakeholders, including provincial and territorial governments, also noted that they lacked a clear “window of access” (contact) when they wanted to initiate engagements with NRCan related to energy and climate change.

Overall, we found that appropriate target audiences have been engaged or reached by the ECCP Program. Figure 6 presents a summary of the categories of key stakeholders and their current level of collaboration with the ECCP Program observed by the evaluation. Moving clockwise around the circle of this collaboration map, we found regular or frequent and formalized engagement with some key stakeholder groups (i.e., the rest of NRCan, other government departments, provincial and territorial governments) and less frequent or formal engagement with others. However, note that this is a simplified presentation. Most categories can be further broken out into sub-groups (e.g., division of “industry” into traditional and emerging clean energy, energy producers and energy consumers), each with differing policy priorities and stakeholder networks. The figure does not identify the intersections among different categories of stakeholder (e.g., Indigenous youth).

The evaluation could not conclude on whether the observed level of collaboration with each stakeholder category is appropriate. While target audiences are defined for some specific initiatives, the ECCP Program as a whole has yet to define its desired level of collaboration with each of category of key stakeholder. For example, while the evaluation identified the relationship with Indigenous organizations as one of cooperation (i.e., where groups may provide each other with information but do not engage in frequent, formal communications or share any decision-making), there is no evidence that would identify whether this as the ECCP Program’s desired level of engagement.

Nonetheless, we did identify potential gaps. For example, policy advice is being increasingly offered to decision-makers from sources such as think tanks, industry and academia and more work is required to engage stakeholders in this area. The ECCP Program is encouraged to continue with recent progress made in increasing its collaboration with these groups. Program staff and external stakeholders also expressed a need to increase engagement in some areas, particularly with those groups that are generally underrepresented in policy development and implementation (i.e., women, youth, and Indigenous peoples) and in the intersection among these groups. The ECCP Program’s Equal by 30 Campaign (Box 5) demonstrates a desire and commitment to make progress in this area, at least as it relates to gender equity.

Box 5: Equal by 30 Campaign

This progress is positive. However, given that engagement is a fundamental consideration for energy and climate change policy, the ECCP Program would benefit from a stakeholder management plan to clearly identify who it wants to engage with, for what purpose or to what level of collaboration, and the mechanisms by which it intends to engage. Besides facilitating relationships with key partners and stakeholders, such a plan would also be a useful benchmark against which to measure progress on the program’s outcomes related to engagement.

Figure 6: ECCP Program Collaboration Map (indicates the current level of engagement of the ECCP Program with its key stakeholders

What we found: Program Delivery

There is effective process management

As noted in the section on Program Design, process management includes having and making effective use of human and financial resources, and available tools (e.g., information technology and data access), as well as adequate leadership direction and support. Theory suggests that a policy function with inadequate inputs is more likely to have a short-term focus on “fighting fires” than developing policy foresights. Risks of inadequate process management include untimely or poor quality policy products. It can also create issues for staff retention, if good quality analysts are alienated by inadequate leadership.

Conditions and contextual factors related to process management are mostly within the control or direct influence of the ECCP Program. While it cannot control human and financial resources (determined as part of departmental budgeting processes) or available tools, it can influence these factors by identifying gaps and upwardly communicating strategies to address its needs. It also has direct control over the way in which the activities it undertakes are prioritized and available resources are allocated to achieve results.

There is evidence of effective process management in response to stretched resources

While the evaluation lacks a strong basis for comparison of resource adequacy in policy programs, we can describe observable trends over the evaluation period. There is some evidence that the ECCP Program may be under-resourced to meet the observed increase in work demands, including the increased need for collaboration, planning and reporting to advance major policy priorities. 

From 2014-15 to 2018-19, total planned expenditures on the ECCP Program increased slightly. The ECCP Program overspent its resources by about 11% (Figure 7). This could indicate an issue with adequate resources or the ability to effectively plan for expenditures, or it could be a symptom of the reactive policy process. For example, the largest over-expenditure (in 2016-17) was likely a result of unanticipated costs related to the Generation Energy national dialogue, as the scope of this initiative was not yet defined at the point that planned spending for 2016-17 was established.

Figure 7: ECCP Program Expenditures, 2014-15 to 2018-19

Effective process management does not necessarily require allocating more resources, but rather necessitates effective use of existing resources. Within the ECCP Program, there is evidence of allocation or re-allocation of resources to address policy priorities. Managers are also aware of staff competencies and appropriately reassign staff or reprioritize work to achieve best results within given timeframes. For example, the Energy Policy Branch re-allocated existing resources to create a task team to coordinate policy, design and logistics for the Clean Energy Ministerial 2019.

However, there are now a large number of new energy and climate change initiatives on the horizon. Given current resources, there may be a need for policy managers to more clearly communicate priorities and trade-offs for analysts. The related impact on the ECCP Program’s human and financial resources should also be considered and integrated as relevant into funding requests. The evaluation noted that funding requests that advance energy and climate change priorities rarely include requests for additional resources for the Energy Policy Branch to support the activities of the ECCP Program.

Effective management of human resources is critical to program delivery

As shown in Figure 8, we also observed that the majority of expenditures (74%) over the evaluation period were on salaries and employee benefit plans (EBP). Effective management of human resources is thus critical to effective program delivery. The adequacy of human resources in a policy program is linked to both the number and the competency of the people engaged, and the effectiveness with which these individuals’ skills are used.

Stakeholders considered the quality of staff skillsets to be a strength of the ECCP Program. We found that competencies that would facilitate the Program’s ability to deliver (e.g., analytical, communication and project management skills) are evident in current staff. Policy analysts were also described as being responsive or adaptable to shifts in priorities and political direction.

At the same time, evidence suggests they are often challenged to get clear direction from management, making it difficult to effectively respond to urgent requests. Coupled with a high workload and short timelines, this may have a negative impact on the ECCP Program’s ability to maintain a robust analytical capacity.

Figure 8: ECCP Program Expenditures by Category, 2014-15 to 2018-19^{Footnote 3}

To help policy analysts gain a better understanding of the needs of their audience, they may benefit from increased opportunities to attend briefings with senior managers (even as an observer). This may be particularly true for junior analysts who would not usually attend such meetings.

The evaluation also identified recruitment and retention of staff as issues for human resource management. While the number and distribution of FTEs appears to be relatively consistent over the evaluation period, there were high levels of staff turnover within these positions. The reasons for this are not entirely clear. One contributing factor may be the mobility of policy analysts in the Economics and Social Science Services (EC) Group, who can easily transfer between positions. However, turnover also includes senior management (EX) positions, with related impacts on management continuity and potential impacts on clarity of direction.

Regardless of the cause, literature indicates that frequent staff transitions require a strong culture of change management to maintain networks, retain corporate knowledge and facilitate the handoff of unfinished work. It also suggests that issues related to staff retention could be mitigated to some degree by stronger talent management, i.e., investments in professional development to support enhancement of competencies and for recognition of effective performance. The extent to which these conditions currently exist within the ECCP Program was not assessed by the evaluation.

There is work in progress to address gaps in data

To be effective, policy analysts also need access to tools such as information technology (including software to facilitate data analysis and communications) and access to data and research to support analysis. We found that there are currently gaps in the data available to enable the ECCP Program to conduct strong economic and energy analysis. In the past several years, NRCan and its federal partners (e.g., ECCC) have been unable to answer important energy-related questions due to insufficient availability or accessibility of quality data and analysis (including raw and interpreted statistics) regarding energy sector competitiveness, the transition to a low-carbon economy, technological changes affecting the workforce, and the achievement of environmental targets and their impacts.  Existing energy information from different sources can be inconsistent, incomplete, untimely and inaccessible, compromising the quality of policy analysis and transparency of decision-making.

This gap is expected to be addressed by the CCEI which, among other objectives, will fill data gaps that would improve the overall quality of available energy information (Box 4).

While the total is unknown, there is evidence that ECCP Program activities result in a large number of “policy products”, i.e., the written and oral outputs of the policy process. More important than the number of outputs is their quality. Literature provides criteria to identify the attributes of effective policy products (Box 6).

Box 6: Attributes of Quality Policy Products

Source: Adapted from OAG Manitoba, 2003.

The most immediately apparent risk of quality issues is policy products being returned for more work, resulting in time delays, cost increases and inefficient use of decision-makers’ time. Poor quality policy products could also present risks to program outcomes, if they result in decision-making that is not evidence-based or lacks policy coordination. Inadequate information on stakeholder preferences, reactions or likely responses to a policy can weaken stakeholder buy-in and create negative backlash. Decision-makers require quality products to reduce uncertainty in selection of the most (cost) effective policy option, and to avoid unknowing repeat of policies that have not worked well in similar contexts.

Most of the conditions related to the quality of policy products are within the control or direct influence of the ECCP Program. Most importantly, product quality can be facilitated by strong process management and an operational environment that sets clear expectations and provides the resources, tools and direction required to produce quality products.

The evaluation did not complete a detailed analysis of the quality of the ECCP Program’s policy products. We also found that the Program receives limited feedback on product quality except in rare instances when there are significant deficiencies. However, the limited evidence available indicates that:

• Published products are effectively communicated. This conclusion is based on a small sample of published products (i.e., the Energy Fact Book, Generation Energy Council Report, and products related to the EMMC 2019).
• Unpublished products do not present significant quality concerns. Few internal stakeholders interviewed noted concerns with the quality of these products. All respondents to the evaluation’s survey of Generation Energy who participated in the convening process indicated that they had the information required to participate effectively. Similarly, all respondents to the evaluation’s survey of the EMMC 2019 had a mostly positive view of the convening process. While the majority (90%) indicated they were missing at least some of the information required to participate effectively, additional comments indicate a challenge in the timely availability of meeting materials, particularly for those who need to travel to attend the conference. The process of releasing these documents is managed by another sector (Strategic Policy and Innovation Sector), and not a direct result of activities under the ECCP.

Specific to this policy space, it is particularly important that policy products also be neutral, evidence-based, and inclusive of diverse perspectives. Generation Energy was noted as a positive example in this regard. Of survey respondents who participated in the Generation Energy dialogue, 90% indicated that their perspectives were adequately or somewhat adequately reflected in the end product. Similarly, internal stakeholders expect the ECCP Program to adequately consider all NRCan perspectives and reflect these in its key messages. However, evidence indicates that energy sector perspectives tended to dominate the messages in some of the ECCP Program’s internal products. Particularly where there were tight timelines for deliverables, representatives from other NRCan sectors expressed concerns over perceived limitations in their ability to add their perspectives or otherwise make changes to relevant products that had already been approved by the former ADM Energy Sector (now ADM LCES and ADM SPPIO).

Our findings related to program relevance underline the importance of consistent and active engagement of stakeholders in the policy process. Stakeholder engagement impacts both the quality and effectiveness of policy products. For example, as noted above, there is a risk that inadequate engagement could result in a lack of stakeholder buy-in or negative backlash in response to a policy, or a lack of policy coordination. Many of the conditions or factors in this area are within the control or direct influence of the ECCP Program, while others (such as the quality of working relationships) can be indirectly influenced via the ECCP Program’s activities for convening and collaboration.

We found that appropriate target audiences have been engaged or reached by the ECCP Program. There is clear evidence of efforts to be inclusive in collection of stakeholder perspectives in some key policy exercises. For example, the Generation Energy dialogues were praised by stakeholders as the “single biggest consultation on energy in the history of Canada” (Figure 9).

Figure 9: Recap of Generation Energy Dialogues, 2017

Regardless, while key stakeholders, such as provincial and territorial governments, are consistently engaged, we found that others (particularly underrepresented groups) may not have an adequate spot at the table. For example, while stakeholders representing youth groups praised the inaugural Youth Forum at the Clean Energy Ministerial 2019 hosted by Canada, they noted this as a rare opportunity for youth to directly engage. There is thus a risk that the ECCP Program is not collecting or conveying all stakeholder perspectives, and/or that opportunities to collect fresh perspectives or new solutions are being missed.

Not all stakeholders have an equal capacity to engage. The evaluation noted specific capacity gaps within territorial governments, Indigenous groups and youth. These capacity gaps could relate to these organizations’ human and financial resources, their literacy related to energy and climate change, lack of knowledge of the policy environment and/or strong working relationships within the issue area. While the ECCP Program does not have any outcomes related to capacity building, there are instances where it has directly influenced stakeholders’ capacity to participate. There is also evidence of ongoing gender-based capacity building in the ECCP Program’s Equal by 30 Campaign (Box 6).

The “right” stakeholders are targeted to receive information

Effective stakeholder engagement also requires the “right” stakeholders to be targeted to receive information, and assumes that they are receptive to the information conveyed. Web analytics confirm that there is a strong demand for the Generation Energy Council Report and Energy Fact Book (Figure 10).

Figure 10: Reach of Select ECCP Products, January 2018 – December 2019

Surveys conducted for the evaluation also provide evidence that key policy products are reaching target audiences (Figure 11). For all products surveyed, a majority of stakeholders indicated that they “learned something new”. This suggests that they were receptive to the information presented. However, there is less evidence of use and uptake of the key messages in these products. Significantly fewer respondents reported having changed their perspectives as a result of this information. While this is not necessarily surprising for products such as the Energy Fact Book, the survey also suggests that few had taken any concrete action to develop or implement strategies for using the information. This suggests a challenge in the causal link between the ECCP Program’s immediate and intermediate outcomes as although access to evidence-based, actor-informed policy products enables informed decisions and discussions, it does not necessarily ensure stakeholder buy-in. Possible conditions and external factors impacting this result are further discussed below.

Figure 11: Knowledge, Use and Uptake of ECCP Policy Products^{Footnote 4}

The ability to extend the reach of policy products’ messages via its targeted stakeholders could also be considered a metric of success. Our survey results indicate that the Energy Fact Book and Generation Energy Council Report are widely discussed and referenced (Figure 11). More research would be required to determine the extent that this is true for other products of ECCP Program. While the survey was less positive for the products of the EMMC 2019, the reason for this may be attributed more to the nature of the survey than the quality of the products. Respondents to this survey included those who participated in the related conference; this is not necessarily the target audience for use of information contained in resulting reports.

Canada’s clean energy or low-carbon energy transition cannot be achieved without collaboration

While the ECCP Program supports the development of policy, it depends on stakeholders outside the Program to implement decisions, policies and practices that will advance its ultimate outcomes. The transformation of Canada’s energy systems required for Canada’s low-carbon energy transition cannot be achieved without collaboration. Given this, it is not surprising that stakeholders reported whether or not they perceived opportunities to collaborate as a critical factor in the use of the key messages in the ECCP Program’s policy products. This suggests that the ECCP Program can thus influence the uptake of key messages by highlighting where opportunities to collaborate may exist.

Effective implementation of the large number of federal initiatives now in progress and on the horizon will require clear governance structures and internal coordination within NRCan and the federal government as a whole. The ECCP Program could also play an important support role in this area.

Internally, we found that uncertainties around program governance created by NRCan’s recent reorganization (October 2019) have likely been mitigated. While it is too soon to comment on its effectiveness, an ADM-level committee has met regularly since May 2020 to promote internal coordination among the three sectors now responsible for tackling NRCan’s energy priorities (LCES, SPPIO and ETS).

The policy environment includes all elements – political, legal, socio-economic, technological and ecological – that affect the decisions, actions and efficacy of those involved in the policy process. Currently, there is evidence of a supportive policy environment. The ECCP Program is aligned with domestic and international priorities for energy and climate change. This relevance motivates decision makers to enact policy in this area and to resource or assist those working towards related policy solutions. There are also domestic and international frameworks in place (e.g., UNFCCC and Pan-Canadian Framework) through which groups can collaborate to develop and implement these solutions, which facilitates the achievement of outcomes. At the same time, while some may be aware of the Energy Policy Branch, the ECCP Program itself does not have high visibility – it is not known or understood by most internal or external stakeholders, making it more difficult for these groups to directly support its work.

While the extent to which the policy environment is supportive of the ECCP Program and its outcomes is outside the control or direct influence of the ECCP Program, it is nevertheless a key factor impacting on program delivery. Theory suggests that there are three streams that drive the policy process: (1) a problem stream (the issue is being faced); (2) a policy stream (possible approaches to addressing the problem); and (3) a political stream (understanding of what would garner support in a given context). As shown in Figure 12, as these streams converge, policy windows are opened, providing opportunity to intervene with policy solutions.

Figure 12: Streams in the Policy Process

The importance of this policy window is illustrated by the example for Generation Energy provided in Figure 13. We found that there was a supportive policy environment to conduct the Generation Energy exercise. Resources were allocated to activities to ensure the process that began in April 2017 was effective. The Generation Energy dialogues were planned to ensure that stakeholders were given the opportunities, capacity and inputs required to engage. The result was broad engagement across all key target groups, enabling stakeholder perspectives to be collected and conveyed. As a result of this process, the ECCP Program’s immediate outcomes were achieved, i.e., the Generation Energy Council Report (June 2018) gives the federal government and stakeholders more information to support decision-making. The engagement has also helped grow awareness and change the perceptions or commitments of stakeholders towards Canada’s energy transition.

Figure 13: Example of Program Delivery for Generation Energy

However, then the policy window closed. The political landscape shifted following elections across a number of provinces, as well as preparations for the federal election that took place in fall 2019. While the pathways indicated by Generation Energy have become a foundation for NRCan’s advice to the government on how to advance Canada’s low-carbon energy transition, these changes to the broader policy environment impacted the ability and space for NRCan to begin implementing its specific recommendations.

There are signs that this is beginning to change – in December 2019, the Minister of Natural Resources’ mandate letter included a priority to “work with partners to implement, as appropriate, the recommendations of the Generation Energy Council Report”. However, as of March 2020, the mechanisms through which Generation Energy will result in intermediate or ultimate outcomes (i.e., use in decision-making, resulting in advancement of energy and climate change priorities) remained unclear. This creates a risk for the achievement of future outcomes as stakeholder awareness creates expectations for use of information. If this use is not realized, stakeholders may disengage.

Within this context, what can the ECCP Program do to maintain a supportive policy environment? There are risks that policy direction in this area could change. Even at present, support is not universal. For example, provincial and territorial policies for energy and climate change do not all align with those of the federal government, and there is uncertainty over the direction of some of Canada’s key trading partners. Even when there is clear support for certain policy proposals, the decision to implement a policy is ultimately in the hands of the government. There is thus a clear need for dialogue to reduce tensions over Canada’s energy future, informed by increased knowledge and literacy about potential options for the energy transition, including the associated trade-offs and impacts for each option. Evidence suggests that the ECCP Program’s influence in this area may be maximized by continued production of policy products that are timely and relevant to stakeholder needs for decision-making. Persuasive evidence that decisions, policies or practices will be effective may also be a critical factor in uptake.

The presentation of sound policy options also requires honest input. Being too heavily swayed by the political sphere’s desired messages can impede the quality of policy products and result in a lack of fresh thinking. Consistent with comparable policy programs, the ECCP Program was noted to be highly reactive to changes in political direction. However, while responsive to its target audience, the evaluation found no indications of the program having compromised its integrity or objectivity. Ensuring this ongoing neutrality of products is critical to maintaining the confidence of stakeholders and the effectiveness of the dialogue on energy and climate change.

The ability of the ECCP Program to contribute to outcomes assumes stakeholders favour the information in its policy products over competing sources. For the ECCP Program’s outcomes to be achieved, stakeholders must also identify a real or perceived advantage to be gained from adopting a new policy, decision or practice based on the information it conveys (Figure 14). While these conditions are mostly outside the influence of the program, decisions can be affected by the quality of the evidence and persuasiveness of arguments advanced in the ECCP Program’s policy products. We lack sufficient data to conclude on the extent to which the ECCP Program’s products meet these conditions and are actually favoured in decision-making.

Figure 14: Factors in Deliberate Non-Use of Policy Products

Source: K. Skinner (2007)

The ECCP Program’s ultimate outcome also requires it to simultaneously advance Canada’s priorities related to both energy and climate change. However, there is an inherent tension between these two priorities given that most of Canada’s GHG emissions result from energy production and use. Decisions taken to reduce emissions must recognize the social and economic importance of resource development, and vice-versa. The ECCP Program’s ultimate outcome thus requires finding opportunities for balanced trade-offs or win-win solutions, i.e., where actions taken to advance priorities for one outcome do not significantly impede progress on the other. Program representatives also note that the sustainable development and competitiveness of energy resources and advancement of climate change objectives are not mutually exclusive, and the ECCP Program aims to bring the two together.

While outcomes achievement (impact) was not the focus of the evaluation, we observed that the ECCP Program lacks strong metrics of its performance in this area. The federal government’s horizontal reports on related priorities (e.g., under the Pan-Canadian Framework, Federal Sustainable Development Strategy, and UN Sustainable Development Goals) provide some indication of the extent to which outcomes are being achieved but fall short of being a scorecard for the ECCP Program.

Besides reviewing actual performance, the evaluation also identified alternative approaches that could be considered as options for program delivery. These approaches are used in other policy functions, both within and outside the field of energy and climate change. Alternative approaches include:

• Use of dedicated policy entrepreneurs embedded in the ECCP Program to lead foresight activities. Policy entrepreneurs are most active in the policy stream (see Figure 12), taking advantage of opportunities to influence policy outcomes to create solutions to potential problems and bring them into the agenda setting process. This could include use of an issues scanning team to proactively identify emerging issues weak signals.
• Use of policy fellows. There are many different kinds of policy fellows. For example, the Alberta-based Energy Futures Lab engages policy fellows as “innovators and influencers”. These individuals are not part of the organization but contribute regularly to its events and policy consultations, and agree to share the Lab’s key messages within their own networks. In other instances, fellowships may involve a short-term placement from outside an organization for the purposes of capacity building, or to foster relationships between researchers and policy makers. These opportunities can provide experts and early-stage professionals interested in policy the opportunity to contribute to the public dialogue, and explore future academic and professional interests.
• Use of external resources to lead analysis and engagements (via contracts or contribution agreements) can add value in provision of technical expertise and perception of neutrality.
• Think tanks can supplement or complement the work of the ECCP Program with independent and expert-driven analysis. For example, with financial support from ECCC, the Canadian Institute for Climate Choices was recently established as an independent, pan-Canadian collaboration working to fill existing information gaps and build a strong, neutral and evidence-based foundation for decision-making on clean growth and climate change. Its work includes research, analysis and extensive engagement with a national network of experts from a range of disciplines to produce and share advice and recommendations on developing integrated policy solutions.
• A normalized process or schedule for multi-stakeholder engagements. We found that the formal engagements currently coordinated by the ECCP Program (e.g., the annual EMMC Conference) do not necessarily include a seat at the table for all target stakeholder groups. While a regular exercise on the scale of Generation Energy is likely neither desirable nor feasible, a normalized process or schedule for multi-stakeholder engagements could provide assurance to key stakeholders that their views will be periodically collected and conveyed. For example, under the Parks Canada Agency Act, the Minister responsible for Parks Canada is required to convene a round table every two years to seek advice from Canadians on matters for which the Agency is responsible. While the format varies, these round tables regularly engage a range of public, not-for-profit and private sector organizations, often with special attention to include underrepresented groups, to provide strategic insights and actionable recommendations on key policy issues.

We lack sufficient evidence from other policy programs to conclude on the effectiveness of these alternatives. However, we did find that some (e.g., use of external resources to lead analysis and engagements) have already been used for specific ECCP Program initiatives while others (e.g., dedicated engagement of policy entrepreneurs) are already in place elsewhere within NRCan with reported success.

Conclusions

The objective of this formative evaluation was to examine the design and delivery of the ECCP Program. The Energy Policy Branch’s understanding of the ECCP Program’s design continues to evolve and improve. Work to finalize the presentation of the ECCP Program’s logic, including articulation of its expected outcomes, is still in progress. Since 2018, the ECCP Program has been undertaking substantive efforts to improve its PIP, including the program description, results logic and performance metrics. Our analysis expands on progress to date to further the development of the theory of change for the ECCP Program. The results of this evaluation should help inform the design of the ECCP Program as it is updated by the Low Carbon Energy Sector. The evaluation also provides insights that can be applied to improve the design and delivery of other policy functions at NRCan.

Energy and climate change are clearly relevant to to federal government and NRCan priorities. Recent updates to the Program’s PIP effected by the Energy Policy Branch better recognize the value of this policy function in addressing these issues. For example, convening is now recognized as critical to the results of the ECCP Program. This is a fundamental shift in program design. Particularly given shared jurisdiction for energy and climate change, Canada’s low-carbon energy transition cannot be achieved without collaboration and stakeholder engagement is critical to the effectiveness of the ECCP Program.

This evaluation reviewed the extent to which the conditions and external factors identified in the theory of change are being met by the ECCP Program and/or are otherwise impacting program delivery. Overall, there is evidence of effective program delivery but  with room for improvement. Across the five categories included in the theory of change, we found that:

• Process Management: There is evidence of effective process management in response to stretched resources, with human and financial resources allocated as required to address priorities. However, these resources may not be aligned with current or future work demands given the increasing number and scope of federal energy and climate change initiatives. Evidence suggests that the Energy Policy Branch could be more effective at securing durable funding to support these initiatives. While competency of existing staff is considered a strength of the program, there have been issues with recruitment and retention of staff including for senior management (EX) positions. The reasons for this are not entirely clear.
• Quality of Policy Products: There is evidence that the ECCP Program produces quality policy products that are neutral and evidence-based. This is critical to meaningful dialogue on energy and climate change. There may be some opportunities for the Program to better reflect the perspectives of other NRCan sectors in its internal products. However, the subject matter is complex and requires a balance of interests to ensure policy products reflect those sometime divergent perspectives. There is also evidence that the ECCP Program produces quality policy products and that these are reaching target audiences. External factors (e.g., political relationships, stakeholder capacity, etc.) impact the extent to which these products influence stakeholder perspectives or result in concrete actions. The ECCP Program is actively working to launch the Canadian Centre on Energy Information, which is expected to address current gaps in access to reliable, transparent and science-based information as the evidence base to make informed decisions. This should enable both Canadians and the ECCP Program to conduct stronger economic and energy analysis.
• Stakeholder Engagement: Overall, we found that the Energy Policy Branch works to engage or reach appropriate target audiences, and to be inclusive of stakeholder perspectives in key policy exercises. While it was not possible to fully assess the adequacy of these collaborations, the evaluation did identify a need to increase engagement in some areas, particularly with those groups that are generally underrepresented in policy development and implementation (i.e., women, youth, and Indigenous peoples). The ECCP Program’s Equal by 30 Campaign is noted as a positive example in this regard. As illustrated by Generation Energy, we also found that engagement is not an end in itself. While the pathways indicated by Generation Energy have become a foundation for NRCan’s advice to the government on how to advance Canada’s low-carbon energy transition, changes to the broader policy environment impacted NRCan’s ability to begin implementing its specific recommendations. Stakeholders need to see concrete actions or there is a risk that they may disengage.
• Policy Environment: The extent to which the policy environment is supportive of the ECCP Program and its outcomes is a key external factor impacting on program delivery. As with all policy programs, policy development within the ECCP Program is largely influenced by government priorities, which can change or evolve quickly depending on the environment.
• Relative Advantage: The ECCP Program can only contribute to its outcome of advancing Canada’s low-carbon energy and climate change priorities if stakeholders use its advice and analysis in decision-making. This also assumes that there is a real or perceived advantage to be gained from adopting the resulting policy, decision or practice. The ECCP Program currently plays an important role in understanding and integrating stakeholders’ polarized views to develop responsive policy options or solutions. Moving forward, its policy advice must continue to recognize the inherent tension between priorities for energy and the environment and actively identify opportunities to bring the two together.

As required by the TB Standard on Evaluation, this report includes clear, actionable recommendations aimed at addressing the key issues or concerns identified by the evaluation. However, the evaluation also recognizes that all conditions and contextual factors – including those that cannot be directly controlled or influenced – will impact program effectiveness. We encourage the ECCP Program to consider all of the conditions and external factors in the theory of change presented in this evaluation report as it moves forward with reviewing and revising its program design.

Appendix 1: Theory of Change for the ECCP Program (Policy is not linear)

Text version

APPENDIX 1 THEORY OF CHANGE FOR THE ECCP PROGRAM (POLICY IS NOT LINEAR).  This figure presents a theory of change for the ECCP Program with arrows that show the direction of linkages among elements of the logic model that was presented in Figure 2. Inputs (dollars, FTEs and tools) feed into all of the model’s activities, outputs and outcomes. A box surrounds the activities and outputs, as elements of the logic model that feed into the program’s professional service function (which creates an evidence base). Activities feed into not only outputs but also to other activities. A larger box representing the program’s convening function (which facilitates feedback, buy-in and collaboration) regroups the activities and outputs with immediate outcomes. A final large box representing the program’s influence and oversight function (which frames policy and influences systemic changes via evidence and constructive dialogue) regroups all of the programs activities, outputs and immediate, intermediate and ultimate outcomes. A dashed line underneath the diagram, labelled revision of problem, issue and/or opportunity, presents a feedback loop that connects the intermediate outcomes for use of information back to the activities. One rectangular element labelled external factors (for example, supportive internal, domestic and international policy environment), underlies all phases of the logic model.

The following table explains the causal links within this theory of change, and outlines the conditions and factor that influence the effective and efficient achievement of related outcomes (where [C] = control, [DI] – direct influence, and [I] = indirect or outside of influence).

Transition in Results Chain	Conditions and Factors that Influence Results Achievement
	Process Management	Quality of Policy Products	Stakeholder Engagement	Relative Advantage
Inputs to Activities	There is adequate leadership direction and support. [DI] There are adequate inputs (i.e., human resources, financial resources, and support tools). [DI]	N/A	N/A	N/A
Causal Link(s): Inputs are used to support and guide the ECCP Program’s activities.
Activities to Outputs	Analysts have access to the inputs (direction, resources and tools) required to complete analysis. [DI] The human resources applied closely fit the skill set required. [C] Analysts are responsive to leadership direction. [C]	Analysts are responsive to stakeholder perspectives. [C]	The “right” stakeholders are engaged to provide perspectives (convening is sufficiently inclusive). [DI] Stakeholders have the capacity to engage. [DI] There are good working relationships (strong networks) among stakeholders. [I]	N/A
Causal Link(s): Research, analysis, coordination and collaboration creates an evidence base for quality policy products; communication of key messages encourages their uptake.
Outputs to Immediate Outcomes	The problems, issues and opportunities to be addressed have been properly identified and clearly defined. [C]	Policy products clearly communicate good quality, relevant and current information. [C] Outputs are distributed in a timely manner. [C]	Stakeholder needs and perspectives are appropriately understood and communicated. [C] The “right” stakeholders are targeted to receive outputs. [C] Targeted stakeholders read, understand and are receptive to the information provided. [DI]	Stakeholders favour the messages in outputs over competing information / perspectives. [I]
Causal Link(s): Access to evidence-based, actor-informed policy products enables informed decisions and discussions, and encourages stakeholder buy-in.
Immediate Outcomes to Intermediate Outcomes	N/A	Key messages are relevant and timely to decisions to be made. [I] There is evidence that the decision, policy or practice will be effective. [I]	Stakeholders identify opportunities to collaborate on implementation of decisions, policies or practices. [I]	There is a real or perceived advantage to advance related decisions, policies or practices. [I]
Causal Link(s): Decision makers use information, advice and analysis to adopt decisions, policies and practices that advance energy and climate change priorities.
Intermediate Outcomes to Ultimate Outcomes	N/A	N/A	N/A	There are opportunities for balanced trade-offs (or win-win scenarios) on energy and climate change. [I]

Appendix 2: Survey methodology and limitations

Methodology

The stakeholder surveys administered for this evaluation were adapted from the Knowledge, Take-up and Use Assessment Tool (KUUT) developed by Dr. Kelly Skinner (University of Waterloo, Faculty of Applied Health Sciences). This quantitative tool has been specifically designed, based on extensive research, to measure the reach and uptake of outputs produced / delivered by different interventions. The KUUT uses a standard questionnaire to measure the reach and uptake of a product, with questions moving sequentially through levels of possible use (Figure 1). It also includes questions to categorize reasons for deliberate non-use. The standard questions in Dr. Skinner’s KUUT were adapted by AEB for the purposes of the evaluation, while adhering as closely as possible to the tool’s intended design.

Figure 1: Progressive Levels of Knowledge, Use and Uptake

Products selected for assessment were identified in consultation with ECCP Program representatives and the evaluation’s technical advisor. In total, this evaluation considers the results of three surveys (see Table 1). One of these, i.e., the survey of the Energy Fact Book, was conducted by the ECCP Program (Energy Policy Branch) as a test of the KUUT as a potential performance metric. While the survey administered by the ECCP Program was also adapted from the KUUT, it used a shortened version of the questionnaire to test only those questions perceived to be of most utility to the Program’s immediate information needs. The two surveys conducted by the AEB also included questions on levels of satisfaction and impact of the convening process amongst targeted stakeholders.

Table 3: Products Surveyed for the Evaluation

Product	Survey Administration
Energy Fact Book	ECCP Program, August 2019
Energy and Mines Ministers’ Conference (EMMC) 2019	AEB, December 2019
Generation Energy Council Report	AEB, December 2019

All three surveys were designed to be administered to a relatively small, targeted set of stakeholders. The specific stakeholders targeted varied by product (see Table 2), with contacts meant to relate to the intended end users of each product, i.e., who the ECCP Program would most expect to access and use the product. In an effort to avoid survey fatigue, evaluators did not send any given stakeholder more than one survey.

Table 4: List of Targeted Survey Stakeholders and Response Rates

Product	Targeted Stakeholders	Total Contacts Reached	Total Responses	Response Rate
Energy Fact Book	Director Generals of NRCan Programs, Statistics Canada, ECCC and Global Affairs Canada	30-33Footnote 5	1	33-37%
Energy and Mines Ministers’ Conference	Working-level provincial and territorial representatives who participated in EMMC 2019	47	11	23%
Generation Energy Council Report	NRCan ADMs, Generation Energy Council, and Generation Energy dialogue participants	71	20	28%

The surveys were each administered using Survey Monkey, AEB’s standard survey software. Surveys were available to respondents in both official languages.

Limitations

A key challenge for the evaluation’s surveys was to identify and reach targeted stakeholders related to each product. In all cases, the specific contacts were provided by the ECCP Program based on available lists. It is possible that this introduced a response bias. In particular, with respect to Generation Energy, we found that e-mail addresses collected with electronic Generation Energy dialogue submissions could not be used for purposes of public opinion research or this survey. Contact lists for more direct participants had also not been adequately maintained given the time elapsed since completion of the dialogues.

Document review and interviews completed during the conduct phase of the evaluation also demonstrated that the ECCP Program’s stakeholders are numerous and varied. As a result, each survey reached a very small sub-set of interested stakeholders. This could indicate a high margin of error in survey results.

While deemed acceptable for the purposes of this analysis, there was a low response rate related to each survey (Table 2). Response rates for the surveys administered by AEB may have been impacted by the timing of the surveys (administered around the holiday period, 11 December 2019 to 6 January 2020). The release of the Minister of Natural Resources’ new mandate letter on December 13, 2020 – with specific commitments related to Generation Energy – may have also introduced some response bias and/or impacted response rates for this survey.

Lastly, analysis of survey results was limited by the nature of the survey tool. The KUUT is not designed to capture input on the content of knowledge products, just reach and uptake. Additional questions to gather more context and/or detailed data on the ways in which information contained in the knowledge products is being used would have added value to survey analysis.     

These challenges and the resulting limitations on survey results could represent important lessons learned should the ECCP Program consider adopting similar surveys as a performance metric.

Appendix 3: List of key documents referenced in report

Colebatch, H. K. (2006). Mapping the Work of Policy, in H. K. Colebatch (Ed.), Beyond the Policy Cycle: The Policy Process in Australia (pp. 1-19).

Environment and Climate Change Canada. (2020). National Inventory Report 1990-2018: Greenhouse Gas Sources and Sinks in Canada.

Generation Energy Council Report (2018). Canada’s Energy Transition: Getting to Our Energy Future, Together. PDF 13.89 MB

Government of Canada. (2016). Pan-Canadian Framework on Clean Growth and Climate Change.

Frey, B. et al. (2006). Measuring Collaboration among Grant Partners, in American Journal of Evaluation (Vol. 27, No. 3).

Kingdon, J.W. (1984). Agendas, Alternatives and Public Policies.

Lemmen, D.S., Johnston, M., Ste-Marie, C. and Pearce, T. (2014). Chapter 3: Natural Resources, in Canada in a Changing Climate: Sector Perspectives on Impacts and Adaptation (pp. 65-98). PDF 2.24 MB

Montague, S. (2014). A Structured Menu for Policy Influence. (unpublished)

Natural Resources Canada. (2019-20). Departmental Plan.

Natural Resources Canada. (2018-19). Departmental Results Framework.

Natural Resources Canada. (2019). Energy Fact Book 2019-2020. PDF 25.1 MB

Office of the Auditor General (OAG) Manitoba. (2003). A Guide to Policy Development. PDF 2.68 KB

Office of the Prime Minister. (2019). Minister of Natural Resources Mandate Letter.

Parsons, W. (2004). Not Just Seeing but Weaving: Relevant Knowledge and the Craft of Building Policy Capacity and Coherence, in Australian Journal of Public Administration, 63, pp. 43-57.

Positive Energy. (2019). Polarization over Energy and Climate in Canada: Survey Results.

Skinner, K. (2007). Developing a Tool to Measure Knowledge Exchange Outcomes, in The Canadian Journal of Program Evaluation (Vol. 22, No. 1), pp. 49-73. PDF 180 KB

Tenbensel, T. (2006). Policy Knowledge for Policy Work, in H. K. Colebatch (Ed.), The Work of Policy: An International Survey (pp. 199-216).

The Council of the Federation. (2015) Canadian Energy Strategy.

Treasury Board of Canada Secretariat. (unpublished draft). Evaluating Policy Functions: Exploring Concepts and Tools.

US Global Change Research Program (USGCRP). (2019). Fourth National Climate Assessment Volume II: Impacts, Risks and Adaptations in the United States.

Appendix 4: Evaluation team

Christian Asselin, (retired) Chief Audit and Evaluation Executive (CAEE)

Michel Gould, Chief Audit and Evaluation Executive (new)

David Ash, Senior Advisor to the CAEE

Stephanie Kalt, Evaluation Manager

Mary Kay Lamarche, Evaluation Manager

Edmund Wolfe, Senior Evaluator

Lauren Omson, Evaluator

Aishwarya Babu, Policy Analyst (PARDP)