Natural Resources Canada Departmental Evaluation Plan for 2017-18 to 2021-22 Final

Natural Resources Canada Departmental Evaluation Plan
2017-18 to 2021-22

Table of Contents

Deputy Head Confirmation Note

I approve the Natural Resource Canada Departmental Evaluation Plan for 2017-18 to 2021-22, which I submit to the Treasury Board of Canada Secretariat as required by the Policy on Results.

I confirm that this five-year rolling Departmental Evaluation Plan:

  • Includes and plans for evaluation of all ongoing programs of grants and contributions with five-year average actual expenditures of $5 million or greater per year at least once every five years, in fulfillment of the requirements of subsection 42.1 of the Financial Administration Act;
  • meets the requirements of the Mandatory Procedures for Evaluation; and
  • supports the requirements of the expenditure management system including, as applicable, memoranda to cabinet, Treasury Board submissions, and resource alignment reviews.

I will ensure that this plan is updated annually, and I will provide information about its implementation to the Treasury Board of Canada Secretariat, as required.

 

Signature: ___________________________________________

                                    Christyne Tremblay
                                      Deputy Minister

Introduction

This document presents Natural Resource Canada’s Departmental Evaluation Plan for 2017-18 to 2021-22. Each year, the Strategic Evaluation Division (SED) within Natural Resource Canada’s (NRCan) Planning, Delivery and Results Branch conducts an evaluation planning exercise to guide the manner in which the Division’s resources are intended to be used for the following 5 fiscal years. The plan serves as a resource for senior management, the Performance Measurement, Evaluation and Experimentation Committee (PMEEC), and the Deputy Ministers to:

  • assess the department’s compliance with the Financial Administration Act (FAA) and the Policy on Results;
  • ensure evaluation commitments in submissions approved by the Treasury Board are respected;
  • confirm evaluation coverage of the areas of highest risk for the department; and
  • assess the adequacy of resourcing for the evaluation function.

To support the Deputy and the PMEEC in overseeing the evaluation function, the current document outlines the policy and operational context within which evaluation services are being delivered at NRCan, describes the approach used to develop the plan, including an analysis of Division resources, and provides detailed information related to the rationale, resources, and timelines for each evaluation planned over the next five years (see Annex A).

Planning Context

Between 2009 and 2016, under the auspices of the Policy on Evaluation, annual evaluation planning largely put aside risk as a consideration for the selection of evaluation projects. Rather, policy requirements for 100% evaluation coverage of the department’s direct program spending, including all ongoing programs of grants and contributions, every five years often meant that all departmental programs were targeted for evaluation, regardless of risk. Other policy exigencies, such as mandatory evaluation issues, generated efficiencies by allowing departments to adopt standard approaches to all evaluations.

In July 2016, the Policy on Results replaced the Policy on Evaluation, the Policy on Management Resources and Results Structures, and the Policy on Reporting of Federal Institutions and Corporate Interests to Treasury Board of Canada Secretariat. To further emphasize and promote transparent, results-oriented and evidence-based approaches to federal policy development and program delivery, the new Policy introduces a number of new requirements, such as:

  • integrated oversight of evaluation and performance measurement functions under a single governance committee to ensure the two activities are mutually supportive;Footnote 1
  • the development of a Departmental Results FrameworkFootnote 2 to replace the strategic outcomes and programs of the current Program Alignment Architecture; and
  • the establishment of core competencies for designated heads of Evaluation and Performance Measurement.

With regard to the direction given for the administration of the evaluation function by federal departments, the Policy on Results differs from its predecessor in a number of respects that have important implications for evaluation planning, including:

  • the removal of mandatory evaluation coverage targets and greater flexibility with regard to requirements to comply with the FAAFootnote 3 , both of which promote a more risk-based approach to planning;
  • the requirement that department-defined evaluations be planned on the basis of need, risk and priority, although everything should be evaluated periodically;
  • the removal of mandatory evaluation issues for non-FAA required evaluations, which is intended to promote greater innovation and customization of research approaches but may also require greater investment in project planning;
  • clarified program accountabilities for the development of mandatory performance information profiles for all departmental programs, which is expected to lead to a greater availability and higher quality of performance measurement information to assess program effectiveness; and
  • the establishment of a program inventory that, starting in 2018-19, will replace the current Program Activity Architecture as the universe against which 5-year evaluation planning is conducted for many federal departments.

During 2017-18, it will be important for the department to position itself to take advantage of these flexibilities. In particular, despite the removal or loosening of several mandatory requirements, the department’s ability to direct its evaluation function continues to be somewhat constrained under a large number of mandatory evaluations (as described in The Overview of Planned Evaluations for 2017-18, below). NRCan will engage with Treasury Board of Canada Secretariat (TBS) over the coming year to identify strategies and approaches to lessening the overall burden of mandatory evaluations related to sunsetting programs.

In addition to the delivery of evaluation services to the department, the Strategic Evaluation Division is responsible for:

  • reviewing memoranda to cabinet and Treasury Board submissions to ensure plans for performance information and evaluations are sufficient and that information on past evaluations is accurately represented and balanced;
  • providing expert advice and services in the areas of research methods, evaluation and performance measurement for program and internal audit initiatives, and to support broader departmental priorities (e.g., innovation and experimentation); and
  • supporting the department’s strategic planning and reporting activities, such as establishing performance measurement frameworks for mandate letter commitments, the creation of a program inventory, and the development of performance information profiles.

The five-year plan must also make provisions to respond to other aspects of TBSoversight.

These mandatory requirements include: reporting on the completion of the previous year’s workplan; completing an annual capacity assessment survey; undertaking a neutral assessment of the evaluation function every five years; and addressing any evaluations requested by the Secretary of the Treasury Board of Canada or required to support centrally-led evaluations or resource alignment reviews.

Planning Approach

The 2017-18 Program Alignment Architecture was used to identify the universe of all possible programs that might be considered for evaluation over the course of the next five years. In selecting programs to be evaluated, priority was first given to all mandatory evaluations, including:

  • all ongoing programs of grants and contributions with average annual expenditures of $5M or more;Footnote 4 and
  • evaluation commitments made as part of Treasury Board submissions.

The plan was developed such that evaluation evidence will be available in time to inform program renewal decisions and to respond to FAA requirements. Once all mandatory evaluations were accommodated within the plan, a risk assessment was undertaken of the remaining programs within the planning universe. The risk assessment exercise involved joint consultations with the Internal Audit Branch to solicit senior management feedback on program risks and priorities for evaluation, as well as a review of departmental documentation (e.g., NRCan’s Corporate Risk Profile, business planning documentation, NRCan’s Report on Plans and Priorities, departmental audit plans). Risk was assessed according to various program characteristics (e.g., upcoming renewal, materiality, visibility), management practices (e.g., structures, delivery complexity, performance measurement), past research findings, and upcoming internal audits. Evaluations were then planned for some of the remaining units of the evaluation universe that did not already have a mandatory requirement according to the availability of evaluation resources and the unit’s relative risk ranking.

The final plan also considers feedback and/or requests from the TBS and makes accommodation for non-project related advisory and other activities to support the department in responding to broader federal initiatives related to experimentation, cross-cutting program reviews (e.g., Innovation, Science), or special studies (e.g., prospective evaluations to support program design). As such, resources were set aside to engage with senior management to identify and undertake studies to respond to their information needs (see “Special Studies” in Appendix A).

Analysis of Planned Evaluation Resources

SEDs planned professional services budget and total staff complement is $210,000 and 9.46 full-time employees, respectively. The team of evaluators translates into 113.5 person-months available to undertake all corporate and division activities. The resources available for projects – person-months and professional services – are summarized in Table 1 below.

The summary demonstrates that there is a minimal difference (i.e., 0.7 person months) between the total resources available (62.3 person months) and planned (63 person months) for 2017-18 projects.

Table 1: Strategic Evaluation Division Resource Forecast for 2017-18
Project Resources Summary 2017-18 Person Months
Resources Available
Total Division Resources (excluding management and support staff) 113.5
Less: Training, Holidays, administrative activities, meetings, and other 51.2
Total Resources Available (person-days) 62.3
Resources Planned
Resources Required per Plans (Appendix A) 63
Professional Services
Project Prof Services per Plan (Appendix A) $210,000

Notes:
1. A person month is a unit of measurement reflecting level of effort. One person month is equal to twenty 7.5 hours days.
2. Total ResourcesAvailablecorresponds to budgeted full-time employees converted to person months and adjusted for non-project time, including meetings, leave (volunteer, vacation, sick), general administration, and advisory services.
3. Resources Requiredis the total number of person months forecasted to complete the 2017-18 evaluation projects, as per the detailed plan presented in Appendix A.
4. Professional Servicesis the budgeted cost of professional services planned in support of projects, as detailed in the plans presented in appendix A.

Completion of Planned Evaluations from 2016-17

According to the 2016-17 joint audit and evaluation plan, a total of 5 NRCan-led evaluations and three horizontal evaluations led by other government departments (OGDs) were scheduled for completion in 2016-17. These included:

  • Forest Disturbances Science and Application
  • Nuclear Legacy Liabilities Program
  • Essential Geographic Information and Support
  • Mining Innovation
  • Class Grants and Contributions Program
  • Horizontal Climate Change Adaptation (led by Environment and Climate Change Canada)
  • Single Window Initiative (led by Canada Border Services Agency)
  • Genomics R&D Initiative (led by the National Research Council)

While the vast majority of the project work for each of the five NRCan evaluations was completed by the end of the fiscal year (i.e., all had revised draft reports as of March 31st, 2017), none received final approval during the fiscal year. The delays in receiving final approvals stemmed from a number of factors, including management turnover, staff attrition, unexpected project delays (e.g., the quality of supplier deliverables), and challenges in transitioning to a new organizational and governance model, as required by the Policy on Results.

Of the three OGD-led evaluations, the Genomics R&D Initiative evaluation was completed, the Horizontal Evaluation of Climate Change Adaptation was not completed, and the Single Window Initiative evaluation was postponed to 2018-19.

Overview of Evaluations Planned for 2017-18

Table 2 presents a summary of all evaluation projects that will be active during the 2017-18 fiscal period. Work will occur on a total of 18 evaluation projects in 2017-18. Of these 18 projects, 11 projects respond to a Treasury Board commitment to evaluate, three respond to both a TB commitment and section 42(1) requirements of the Financial Administration Act, and only four are risk-based.

Ten of the 18 active 2017-18 projects are planned to be completed in 2017-18 and of these, six are projects that were originally intended to be completed in 2016-17 and were carried over to 2017-18 (i.e., 5 NRCan and 1 OGD-led evaluation, as described in the previous section). In terms of leadership on the evaluations, 14 of the 18 projects will be led by NRCan.

Despite the large overall number of evaluations that are expected to be active in 2017-18, internal analyses suggest that this can be accommodated within existing allocations, as the majority of these evaluations (n=13) require only a limited resource investment. Specifically, six are nearly completed carry-over projects, four are evaluations led by other government departments that will undertake the bulk of the work, and four are being delivered by external consultants.

Table 2: Summary of Active Evaluation Projects in 2017-18
Project Title / PAA # Evaluation Type Expected Completion
Evaluations led by NRCan (carried over from 2016-17)
Class Grants and Contributions Program TB requirement Q2 2017-18
Nuclear Legacy Liabilities Program TB requirement Q2 2017-18
Mining Innovation Program (1.2.1) Risk-based Q2 2017-18
Geomapping for Energy and Minerals (1.3.3) TB requirement Q3 2017-18
Major Projects Management Office Initiative (1.3.5) TB requirement Q3 2018-19
EcoEnergy for Biofuels (2.1.3) TB requirement Q3 2017-18
Forest Disturbances Science and Application (3.1.3) Risk-based Q2 2017-18
Essential Geographic Information (3.2.1) Risk-based Q2 2017-18
Evaluations led by NRCan (beginning 2017-18)
Energy Policy Capacity (1.1.3 and 2.1.2) TB / FAA requirement Q2 2018-19
MPMO West / West Coast Energy Initiative (1.1.3) TB requirement Q4 2018-19
Forest Sector Innovation (1.2.2) TB / FAA requirement Q3 2018-19
Mineral Investment (1.3.1) Risk-based Q3 2018-19
Energy Efficiency (2.1.4) TB / FAA requirement Q3 2018-19
Energy Innovation Program (2.2.3) TB requirement Q2 2018-19
Horizontal evaluations – led by other departments – Active in 2017-18
Horizontal Climate Change Adaptation (various) TB requirement led by ECCC Q1 2017-18
Youth Employment Strategy (2.2.3) TB requirement led by ESDC Q2 2019-20
Extended Continental Shelf (3.2.4) TB requirement led by GAC Q4 2017-18
Federal Contaminated Sites (4.1.1.) TB requirement led by ECCC Q2 2018-19

Annex A presents a detailed schedule of evaluations over the next five years, including the evaluation rationale and timing, program resources considered, partners, and resources estimated to complete the evaluation. Also included in the annex are all programs in the PAA that are not scheduled for evaluation within the upcoming 5-year timeframe (i.e., see non-highlighted entries), given limits to Division resources, the absence of a mandatory requirement to evaluate, and their lower relative risks, as described under Planning Approach, above.

It is important to note that starting in 2018-19, the evaluation planning universe will change from the current PAA to the program inventory in the Departmental Results Framework. As such, planned evaluations for 2018-19 and beyond are expected to change somewhat to reflect a risk-based approach and alignment with the new program inventory. Efforts have been made, however, to ensure the new program inventory is fairly well aligned to the current PAA, and so changes to the plan from 2017-18 to 2018-19 are expected to be minimal.


Annex A: Detailed 5-Year Schedule of Evaluations


PAA
Element
Specific
Commitment(s)
Sector Type Lead  Start  Date   End  Date  2017-18 G&C 2017-18 DPS 2017-18 2018-19 PM 2019-20 2020-21 2021-22
$ % $ % PM1 $(000s) PM $(000s) PM $(000s) PM $(000s) PM $(000s)
1.1.1 - Mineral and Metal Markets Access and Diversification 1.1.1 - Mineral and Metal Markets Access and Diversification LMS         50,000 0.02 1,584,455 0.20                    
1.1.2 - Forest Products Market Access and Diversification 1.1.2 - Forest Products Market Access and Diversification CFS G&C   19-20 Q1 20-21 Q4 11,643,000 3.58 28,874,287 3.64         6 40 6      
1.1.3 - Energy Market Access and Diversification Energy Policy Capacity (with 2.1.2) ES TB and G&C   16-17 Q4 18-19 Q2 6,203,500 1.91 26,656,680 3.36 10   4              
1.1.3 - Energy Market Access and Diversification MPMO West / West Coast Energy Initiative ES TB NRCan 17-18 Q2 18-19 Q4 0 0 3,075,175 0.39 2 40 3 60            
1.2.1 - Mining Innovation 1.2.1 - Mining Innovation LMS     15-16 Q3 17-18 Q2 0 0 8,592,587 1.08 1                  
1.2.2 - Forest Sector Innovation 1.2.2 - Forest Sector Innovation CFS TB and G&C   17-18 Q2 18-19 Q3 58,205,728 17.91 92,546,943 11.67 10   4              
1.2.3 - Geospatial Innovation 1.2.3 - Geospatial Innovation SPRS         572,000 0.18 8,015,314 1.01                    
1.3.1 - Mineral Investment 1.3.1 - Mineral Investment LMS R   17-18 Q3 18-19 Q3 0 0 10,202,794 1.29 7   7              
1.3.2 - Targeted Geoscience Initiative 1.3.2 - Targeted Geoscience Initiative LMS R   18-19 Q2 19-20 Q3 651,910 0.20 11,729,315 1.48     5   9 30        
1.3.3 - Geomapping for Energy and Minerals 1.3.3 - Geomapping for Energy and Minerals LMS TB   16-17 Q3 17-18 Q4 863,090 0.27 39,632,451 5 3 50                
1.3.4 - Geoscience for New Energy Supply 1.3.4 - Geoscience for New Energy Supply (see 2.2.3) LMS         0 0 5,857,162 0.74                    
1.3.5 - Major Projects Management Office Initiative 1.3.5 - Major Projects Management Office Initiative MPMO TB NRCan 16-17 Q4 18-19 Q3 1,476,000 0.45 5,741,931 0.72 5 70 2              
1.4 Statutory - Atlantic Offshore             408,998,253 125.88 408,998,253 51.58                    
2.1.1 - Renewable Energy Deployment 2.1.1 - Renewable Energy Deployment ES TB   18-19 Q3 20-21 Q1 119,553,000 36.79 120,787,978 15.23     4 25 6 40 2      
2.1.2 - Support for Clean Energy Decision-Making Energy Policy Capacity (see 1.1.3) ES         738,000 0.23 2,335,641 0.29                    
2.1.2 - Support for Clean Energy Decision-Making Clean Growth and Climate Change - International Cooperation ES TB ECCC 18-19 Q4 20-21 Q2 0 0 3,943,468 0.50     0.50   1   0.50      
2.1.3 - Alternative Transportation Fuels 2.1.3 - Alternative Transportation Fuels ES G&C   20-21 Q2 21-22 Q3 10,990,000 3.38 12,868,654 1.62             5   5  
2.1.3 - Alternative Transportation Fuels

EcoEnergy for Biofuels

Joint audit and evaluation
ES TB   16-17 Q2 17-18 Q3 0 0 0 0 6                  
2.1.4 - Energy Efficiency 2.1.4 - Energy Efficiency ES TB and G&C   17-18 Q2 18-19 Q3 4,540,818 1.40 43,525,805 5.49 5   7 25            
2.2.1 - Materials for Energy 2.2.1 - Materials for Energy LMS         0 0 10,016,515 1.26                    
2.2.2 - Green Mining 2.2.2 - Green Mining LMS R   20-21 Q1 21-22 Q3 0 0 13,007,623 1.64             8   4  
2.2.3 - Clean Energy Science and Technology 2.2.3 - Clean Energy Science and Technology (including 1.3.4) IETS G&C   20-21 Q1 21-22 Q3 85,331,000 26.26 163,634,604 20.64             10   4  
2.2.3 - Clean Energy Science and Technology World-Class Tanker Safety System (Phase II). IETS TB TC 18-19 Q1 18-19 Q4 0 0 2,212,440 0.28     2              
2.2.3 - Clean Energy Science and Technology Energy Innovation Program (Budget 2016) IETS TB   17-18 Q2 18-19 Q2 23,000,000 7.08 29,300,000 3.70 2 50 2 50            
2.2.3 - Clean Energy Science and Technology Youth Employment Strategy (ESDC Lead) OCS TB ESDC 16-17 Q1 19-20 Q2 0 0 1,794,000 0.23 1   1   1          
2.3.1 - Forest Ecosystem Science and Application 2.3.1 - Forest Ecosystem Science and Application CFS         15,000 0 15,605,942 1.97                    
2.3.2 - Groundwater Geoscience 2.3.2 - Groundwater Geoscience LMS         0 0 3,233,122 0.41                    
2.3.3 - Environmental Studies and Assessments 2.3.3 - Environmental Studies and Assessments LMS R   20-21 Q1 21-22 Q2 0 0 7,287,409 0.92             4 50 2  
2.3.4 - Geospatial Information for Responsible Natural Resource Development 2.3.4 - Geospatial Information for Responsible Natural Resource Development SPRS         0 0 1,311,150 0.17                    
3.1.1 - Explosives Safety and Security 3.1.1 - Explosives Safety and Security LMS R   18-19 Q3 20-21 Q1 0 0 4,524,427 0.57     3 25 6 30 1      
3.1.1 - Explosives Safety and Security Single Window Initiative LMS TB CBSA 18-19 Q1 18-19 Q4 0 0 760,000 0.10     2              
3.1.2 - Materials and Certification for Safety and Security 3.1.2 - Materials and Certification for Safety and Security LMS         0 0 1,860,386 0.23                    
3.1.3 - Forest Disturbances Science and Application 3.1.3 - Forest Disturbances Science and Application CFS R   15-16 Q3 17-18 Q2 43,000 0.01 40,735,113 5.14 1                  
3.1.4 - Climate Change Adaptation 3.1.4 - Climate Change Adaptation LMS TB ECCC 19-20 Q3 21-22 Q4 1,000,000 0.31 7,042,592 0.89         1   2   1  
3.1.5 - Geohazards and Public Safety 3.1.5 - Geohazards and Public Safety LMS TB   20-21 Q3 21-22 Q4 35,000 0.01 15,495,561 1.95             5   7  
3.2.1 - Essential Geographic Information 3.2.1 - Essential Geographic Information LMS, SPRS R   15-16 Q2 17-18 Q2 10,000 0 13,226,861 1.67 1                  
3.2.1 - Essential Geographic Information Geomatics - Remote Sensing and Mapping CFS R   18-19 Q3 19-20 Q4 0 0 13,226,861 1.67     4   10 30        
3.2.2 - Canada's Legal Boundaries 3.2.2 - Canada's Legal Boundaries
Joint audit and evaluation
LMS R   18-19 Q3 19-20 Q4 0 0 12,851,311 1.62     4   8 30        
3.2.3 - Polar Continental Shelf Logistics Support 3.2.3 - Polar Continental Shelf Logistics Support LMS         0 0 8,569,697 1.08                    
3.2.4 UN Convention on the Law of the Sea Extended Continental Shelf LMS TB GAC 17-18 Q2 17-18 Q4 0 0 1,275,447 0.16 2                  
4.1.1 Management and Oversight Federal Contaminated Sites Action Plan CMSS TB ECCC 16-17 Q4 18-19 Q2 0 0 3,030,000 0.383 1   1              
Other Nuclear Legacy Liabilities Program4 ES TB   15-16 Q3 17-18 Q2 -- -- -- -- .5                  
Other Class Grants and Contributons5 All TB   15-16 Q3 17-18 Q2 -- -- -- -- .5                  
Other Special Studies All R       -- -- -- -- 5   5   5   5   5  
Other Neutral Assessment       18-19 Q1 18-19 Q4             2 25            
Total             $324,921,046 100% $792,941,701 100% 63 $210 62.50 $210 532 $200 48.52 $50 282 $0

1 PM or person month is a unit of measurement reflecting level of effort. One person month is equal to twenty 7.5 hours days.
2 The latter three years of the plan have purposefully been under-scheduled in order to accommodate the large number of evaluation commitments related to 2017-18 Treasury Board submissions.
3 The proportion of direct program spending accounted for by the FCSAP program is not included in the total, as it is aligned to an internal services sub-program. Total values reflect total program spending excluding internal services.
4 Program has sunset. As such, there is no 2017-18 grants and contribution (G&C) or direct program spending (DPS) associated with this program
5 The value of the class grants and contributions program spending is listed next to each of the sub-programs that make use of these terms and conditions.

 

Legend

Sector:
CFS = Canadian Forest Service
CMSS = Corporate
ES = Energy Sector
IETS = Innovation and Energy Technology Sector
LMS = Lands and Minerals Sector
SPRS = Strategic Policy and Results Sector

Type:
G&C = FAA 42(1) requirement
R = Risk-based
TB = Treasury Board Commitment

Lead:
CBSA = Canada Border Services Agency
ECCC = Environment and Climate Change Canada
ESDC = Employment and Social Development Canada
GAC = Global Affairs Canada
TC = Transport Canada

Colour Coding:

= Active in 2017-18

= Active in 2018-19 or afterward

= Not currently scheduled for evaluation