Natural Resources Canada Departmental Evaluation Plan
2017-18 to 2021-22
Table of Contents
- Deputy Head Confirmation Note
- Planning Context
- Planning Approach
- Completion of Planned Evaluations from 2016-17
- Overview of Evaluations Planned for 2017-18
- Annex A: Detailed 5-Year Schedule of Evaluations
Deputy Head Confirmation Note
I approve the Natural Resource Canada Departmental Evaluation Plan for 2017-18 to 2021-22, which I submit to the Treasury Board of Canada Secretariat as required by the Policy on Results.
I confirm that this five-year rolling Departmental Evaluation Plan:
- Includes and plans for evaluation of all ongoing programs of grants and contributions with five-year average actual expenditures of $5 million or greater per year at least once every five years, in fulfillment of the requirements of subsection 42.1 of the Financial Administration Act;
- meets the requirements of the Mandatory Procedures for Evaluation; and
- supports the requirements of the expenditure management system including, as applicable, memoranda to cabinet, Treasury Board submissions, and resource alignment reviews.
I will ensure that this plan is updated annually, and I will provide information about its implementation to the Treasury Board of Canada Secretariat, as required.
This document presents Natural Resource Canada’s Departmental Evaluation Plan for 2017-18 to 2021-22. Each year, the Strategic Evaluation Division (SED) within Natural Resource Canada’s (NRCan) Planning, Delivery and Results Branch conducts an evaluation planning exercise to guide the manner in which the Division’s resources are intended to be used for the following 5 fiscal years. The plan serves as a resource for senior management, the Performance Measurement, Evaluation and Experimentation Committee (PMEEC), and the Deputy Ministers to:
- assess the department’s compliance with the Financial Administration Act (FAA) and the Policy on Results;
- ensure evaluation commitments in submissions approved by the Treasury Board are respected;
- confirm evaluation coverage of the areas of highest risk for the department; and
- assess the adequacy of resourcing for the evaluation function.
To support the Deputy and the PMEEC in overseeing the evaluation function, the current document outlines the policy and operational context within which evaluation services are being delivered at NRCan, describes the approach used to develop the plan, including an analysis of Division resources, and provides detailed information related to the rationale, resources, and timelines for each evaluation planned over the next five years (see Annex A).
Between 2009 and 2016, under the auspices of the Policy on Evaluation, annual evaluation planning largely put aside risk as a consideration for the selection of evaluation projects. Rather, policy requirements for 100% evaluation coverage of the department’s direct program spending, including all ongoing programs of grants and contributions, every five years often meant that all departmental programs were targeted for evaluation, regardless of risk. Other policy exigencies, such as mandatory evaluation issues, generated efficiencies by allowing departments to adopt standard approaches to all evaluations.
In July 2016, the Policy on Results replaced the Policy on Evaluation, the Policy on Management Resources and Results Structures, and the Policy on Reporting of Federal Institutions and Corporate Interests to Treasury Board of Canada Secretariat. To further emphasize and promote transparent, results-oriented and evidence-based approaches to federal policy development and program delivery, the new Policy introduces a number of new requirements, such as:
- integrated oversight of evaluation and performance measurement functions under a single governance committee to ensure the two activities are mutually supportive;Footnote 1
- the development of a Departmental Results FrameworkFootnote 2 to replace the strategic outcomes and programs of the current Program Alignment Architecture; and
- the establishment of core competencies for designated heads of Evaluation and Performance Measurement.
With regard to the direction given for the administration of the evaluation function by federal departments, the Policy on Results differs from its predecessor in a number of respects that have important implications for evaluation planning, including:
- the removal of mandatory evaluation coverage targets and greater flexibility with regard to requirements to comply with the FAAFootnote 3 , both of which promote a more risk-based approach to planning;
- the requirement that department-defined evaluations be planned on the basis of need, risk and priority, although everything should be evaluated periodically;
- the removal of mandatory evaluation issues for non-FAA required evaluations, which is intended to promote greater innovation and customization of research approaches but may also require greater investment in project planning;
- clarified program accountabilities for the development of mandatory performance information profiles for all departmental programs, which is expected to lead to a greater availability and higher quality of performance measurement information to assess program effectiveness; and
- the establishment of a program inventory that, starting in 2018-19, will replace the current Program Activity Architecture as the universe against which 5-year evaluation planning is conducted for many federal departments.
During 2017-18, it will be important for the department to position itself to take advantage of these flexibilities. In particular, despite the removal or loosening of several mandatory requirements, the department’s ability to direct its evaluation function continues to be somewhat constrained under a large number of mandatory evaluations (as described in The Overview of Planned Evaluations for 2017-18, below). NRCan will engage with Treasury Board of Canada Secretariat (TBS) over the coming year to identify strategies and approaches to lessening the overall burden of mandatory evaluations related to sunsetting programs.
In addition to the delivery of evaluation services to the department, the Strategic Evaluation Division is responsible for:
- reviewing memoranda to cabinet and Treasury Board submissions to ensure plans for performance information and evaluations are sufficient and that information on past evaluations is accurately represented and balanced;
- providing expert advice and services in the areas of research methods, evaluation and performance measurement for program and internal audit initiatives, and to support broader departmental priorities (e.g., innovation and experimentation); and
- supporting the department’s strategic planning and reporting activities, such as establishing performance measurement frameworks for mandate letter commitments, the creation of a program inventory, and the development of performance information profiles.
The five-year plan must also make provisions to respond to other aspects of TBSoversight.
These mandatory requirements include: reporting on the completion of the previous year’s workplan; completing an annual capacity assessment survey; undertaking a neutral assessment of the evaluation function every five years; and addressing any evaluations requested by the Secretary of the Treasury Board of Canada or required to support centrally-led evaluations or resource alignment reviews.
The 2017-18 Program Alignment Architecture was used to identify the universe of all possible programs that might be considered for evaluation over the course of the next five years. In selecting programs to be evaluated, priority was first given to all mandatory evaluations, including:
- all ongoing programs of grants and contributions with average annual expenditures of $5M or more;Footnote 4 and
- evaluation commitments made as part of Treasury Board submissions.
The plan was developed such that evaluation evidence will be available in time to inform program renewal decisions and to respond to FAA requirements. Once all mandatory evaluations were accommodated within the plan, a risk assessment was undertaken of the remaining programs within the planning universe. The risk assessment exercise involved joint consultations with the Internal Audit Branch to solicit senior management feedback on program risks and priorities for evaluation, as well as a review of departmental documentation (e.g., NRCan’s Corporate Risk Profile, business planning documentation, NRCan’s Report on Plans and Priorities, departmental audit plans). Risk was assessed according to various program characteristics (e.g., upcoming renewal, materiality, visibility), management practices (e.g., structures, delivery complexity, performance measurement), past research findings, and upcoming internal audits. Evaluations were then planned for some of the remaining units of the evaluation universe that did not already have a mandatory requirement according to the availability of evaluation resources and the unit’s relative risk ranking.
The final plan also considers feedback and/or requests from the TBS and makes accommodation for non-project related advisory and other activities to support the department in responding to broader federal initiatives related to experimentation, cross-cutting program reviews (e.g., Innovation, Science), or special studies (e.g., prospective evaluations to support program design). As such, resources were set aside to engage with senior management to identify and undertake studies to respond to their information needs (see “Special Studies” in Appendix A).
Analysis of Planned Evaluation Resources
SEDs planned professional services budget and total staff complement is $210,000 and 9.46 full-time employees, respectively. The team of evaluators translates into 113.5 person-months available to undertake all corporate and division activities. The resources available for projects – person-months and professional services – are summarized in Table 1 below.
The summary demonstrates that there is a minimal difference (i.e., 0.7 person months) between the total resources available (62.3 person months) and planned (63 person months) for 2017-18 projects.
|Project Resources Summary 2017-18||Person Months|
|Total Division Resources (excluding management and support staff)||113.5|
|Less: Training, Holidays, administrative activities, meetings, and other||51.2|
|Total Resources Available (person-days)||62.3|
|Resources Required per Plans (Appendix A)||63|
|Project Prof Services per Plan (Appendix A)||$210,000|
1. A person month is a unit of measurement reflecting level of effort. One person month is equal to twenty 7.5 hours days.
2. Total ResourcesAvailablecorresponds to budgeted full-time employees converted to person months and adjusted for non-project time, including meetings, leave (volunteer, vacation, sick), general administration, and advisory services.
3. Resources Requiredis the total number of person months forecasted to complete the 2017-18 evaluation projects, as per the detailed plan presented in Appendix A.
4. Professional Servicesis the budgeted cost of professional services planned in support of projects, as detailed in the plans presented in appendix A.
Completion of Planned Evaluations from 2016-17
According to the 2016-17 joint audit and evaluation plan, a total of 5 NRCan-led evaluations and three horizontal evaluations led by other government departments (OGDs) were scheduled for completion in 2016-17. These included:
- Forest Disturbances Science and Application
- Nuclear Legacy Liabilities Program
- Essential Geographic Information and Support
- Mining Innovation
- Class Grants and Contributions Program
- Horizontal Climate Change Adaptation (led by Environment and Climate Change Canada)
- Single Window Initiative (led by Canada Border Services Agency)
- Genomics R&D Initiative (led by the National Research Council)
While the vast majority of the project work for each of the five NRCan evaluations was completed by the end of the fiscal year (i.e., all had revised draft reports as of March 31st, 2017), none received final approval during the fiscal year. The delays in receiving final approvals stemmed from a number of factors, including management turnover, staff attrition, unexpected project delays (e.g., the quality of supplier deliverables), and challenges in transitioning to a new organizational and governance model, as required by the Policy on Results.
Of the three OGD-led evaluations, the Genomics R&D Initiative evaluation was completed, the Horizontal Evaluation of Climate Change Adaptation was not completed, and the Single Window Initiative evaluation was postponed to 2018-19.
Overview of Evaluations Planned for 2017-18
Table 2 presents a summary of all evaluation projects that will be active during the 2017-18 fiscal period. Work will occur on a total of 18 evaluation projects in 2017-18. Of these 18 projects, 11 projects respond to a Treasury Board commitment to evaluate, three respond to both a TB commitment and section 42(1) requirements of the Financial Administration Act, and only four are risk-based.
Ten of the 18 active 2017-18 projects are planned to be completed in 2017-18 and of these, six are projects that were originally intended to be completed in 2016-17 and were carried over to 2017-18 (i.e., 5 NRCan and 1 OGD-led evaluation, as described in the previous section). In terms of leadership on the evaluations, 14 of the 18 projects will be led by NRCan.
Despite the large overall number of evaluations that are expected to be active in 2017-18, internal analyses suggest that this can be accommodated within existing allocations, as the majority of these evaluations (n=13) require only a limited resource investment. Specifically, six are nearly completed carry-over projects, four are evaluations led by other government departments that will undertake the bulk of the work, and four are being delivered by external consultants.
|Project Title / PAA #||Evaluation Type||Expected Completion|
|Evaluations led by NRCan (carried over from 2016-17)|
|Class Grants and Contributions Program||TB requirement||Q2 2017-18|
|Nuclear Legacy Liabilities Program||TB requirement||Q2 2017-18|
|Mining Innovation Program (1.2.1)||Risk-based||Q2 2017-18|
|Geomapping for Energy and Minerals (1.3.3)||TB requirement||Q3 2017-18|
|Major Projects Management Office Initiative (1.3.5)||TB requirement||Q3 2018-19|
|EcoEnergy for Biofuels (2.1.3)||TB requirement||Q3 2017-18|
|Forest Disturbances Science and Application (3.1.3)||Risk-based||Q2 2017-18|
|Essential Geographic Information (3.2.1)||Risk-based||Q2 2017-18|
|Evaluations led by NRCan (beginning 2017-18)|
|Energy Policy Capacity (1.1.3 and 2.1.2)||TB / FAA requirement||Q2 2018-19|
|MPMO West / West Coast Energy Initiative (1.1.3)||TB requirement||Q4 2018-19|
|Forest Sector Innovation (1.2.2)||TB / FAA requirement||Q3 2018-19|
|Mineral Investment (1.3.1)||Risk-based||Q3 2018-19|
|Energy Efficiency (2.1.4)||TB / FAA requirement||Q3 2018-19|
|Energy Innovation Program (2.2.3)||TB requirement||Q2 2018-19|
|Horizontal evaluations – led by other departments – Active in 2017-18|
|Horizontal Climate Change Adaptation (various)||TB requirement led by ECCC||Q1 2017-18|
|Youth Employment Strategy (2.2.3)||TB requirement led by ESDC||Q2 2019-20|
|Extended Continental Shelf (3.2.4)||TB requirement led by GAC||Q4 2017-18|
|Federal Contaminated Sites (4.1.1.)||TB requirement led by ECCC||Q2 2018-19|
Annex A presents a detailed schedule of evaluations over the next five years, including the evaluation rationale and timing, program resources considered, partners, and resources estimated to complete the evaluation. Also included in the annex are all programs in the PAA that are not scheduled for evaluation within the upcoming 5-year timeframe (i.e., see non-highlighted entries), given limits to Division resources, the absence of a mandatory requirement to evaluate, and their lower relative risks, as described under Planning Approach, above.
It is important to note that starting in 2018-19, the evaluation planning universe will change from the current PAA to the program inventory in the Departmental Results Framework. As such, planned evaluations for 2018-19 and beyond are expected to change somewhat to reflect a risk-based approach and alignment with the new program inventory. Efforts have been made, however, to ensure the new program inventory is fairly well aligned to the current PAA, and so changes to the plan from 2017-18 to 2018-19 are expected to be minimal.
Annex A: Detailed 5-Year Schedule of Evaluations
|Sector||Type||Lead||Start Date||End Date||2017-18 G&C||2017-18 DPS||2017-18||2018-19 PM||2019-20||2020-21||2021-22|
|1.1.1 - Mineral and Metal Markets Access and Diversification||1.1.1 - Mineral and Metal Markets Access and Diversification||LMS||50,000||0.02||1,584,455||0.20|
|1.1.2 - Forest Products Market Access and Diversification||1.1.2 - Forest Products Market Access and Diversification||CFS||G&C||19-20 Q1||20-21 Q4||11,643,000||3.58||28,874,287||3.64||6||40||6|
|1.1.3 - Energy Market Access and Diversification||Energy Policy Capacity (with 2.1.2)||ES||TB and G&C||16-17 Q4||18-19 Q2||6,203,500||1.91||26,656,680||3.36||10||4|
|1.1.3 - Energy Market Access and Diversification||MPMO West / West Coast Energy Initiative||ES||TB||NRCan||17-18 Q2||18-19 Q4||0||0||3,075,175||0.39||2||40||3||60|
|1.2.1 - Mining Innovation||1.2.1 - Mining Innovation||LMS||15-16 Q3||17-18 Q2||0||0||8,592,587||1.08||1|
|1.2.2 - Forest Sector Innovation||1.2.2 - Forest Sector Innovation||CFS||TB and G&C||17-18 Q2||18-19 Q3||58,205,728||17.91||92,546,943||11.67||10||4|
|1.2.3 - Geospatial Innovation||1.2.3 - Geospatial Innovation||SPRS||572,000||0.18||8,015,314||1.01|
|1.3.1 - Mineral Investment||1.3.1 - Mineral Investment||LMS||R||17-18 Q3||18-19 Q3||0||0||10,202,794||1.29||7||7|
|1.3.2 - Targeted Geoscience Initiative||1.3.2 - Targeted Geoscience Initiative||LMS||R||18-19 Q2||19-20 Q3||651,910||0.20||11,729,315||1.48||5||9||30|
|1.3.3 - Geomapping for Energy and Minerals||1.3.3 - Geomapping for Energy and Minerals||LMS||TB||16-17 Q3||17-18 Q4||863,090||0.27||39,632,451||5||3||50|
|1.3.4 - Geoscience for New Energy Supply||1.3.4 - Geoscience for New Energy Supply (see 2.2.3)||LMS||0||0||5,857,162||0.74|
|1.3.5 - Major Projects Management Office Initiative||1.3.5 - Major Projects Management Office Initiative||MPMO||TB||NRCan||16-17 Q4||18-19 Q3||1,476,000||0.45||5,741,931||0.72||5||70||2|
|1.4 Statutory - Atlantic Offshore||408,998,253||125.88||408,998,253||51.58|
|2.1.1 - Renewable Energy Deployment||2.1.1 - Renewable Energy Deployment||ES||TB||18-19 Q3||20-21 Q1||119,553,000||36.79||120,787,978||15.23||4||25||6||40||2|
|2.1.2 - Support for Clean Energy Decision-Making||Energy Policy Capacity (see 1.1.3)||ES||738,000||0.23||2,335,641||0.29|
|2.1.2 - Support for Clean Energy Decision-Making||Clean Growth and Climate Change - International Cooperation||ES||TB||ECCC||18-19 Q4||20-21 Q2||0||0||3,943,468||0.50||0.50||1||0.50|
|2.1.3 - Alternative Transportation Fuels||2.1.3 - Alternative Transportation Fuels||ES||G&C||20-21 Q2||21-22 Q3||10,990,000||3.38||12,868,654||1.62||5||5|
|2.1.3 - Alternative Transportation Fuels||
EcoEnergy for BiofuelsJoint audit and evaluation
|ES||TB||16-17 Q2||17-18 Q3||0||0||0||0||6|
|2.1.4 - Energy Efficiency||2.1.4 - Energy Efficiency||ES||TB and G&C||17-18 Q2||18-19 Q3||4,540,818||1.40||43,525,805||5.49||5||7||25|
|2.2.1 - Materials for Energy||2.2.1 - Materials for Energy||LMS||0||0||10,016,515||1.26|
|2.2.2 - Green Mining||2.2.2 - Green Mining||LMS||R||20-21 Q1||21-22 Q3||0||0||13,007,623||1.64||8||4|
|2.2.3 - Clean Energy Science and Technology||2.2.3 - Clean Energy Science and Technology (including 1.3.4)||IETS||G&C||20-21 Q1||21-22 Q3||85,331,000||26.26||163,634,604||20.64||10||4|
|2.2.3 - Clean Energy Science and Technology||World-Class Tanker Safety System (Phase II).||IETS||TB||TC||18-19 Q1||18-19 Q4||0||0||2,212,440||0.28||2|
|2.2.3 - Clean Energy Science and Technology||Energy Innovation Program (Budget 2016)||IETS||TB||17-18 Q2||18-19 Q2||23,000,000||7.08||29,300,000||3.70||2||50||2||50|
|2.2.3 - Clean Energy Science and Technology||Youth Employment Strategy (ESDC Lead)||OCS||TB||ESDC||16-17 Q1||19-20 Q2||0||0||1,794,000||0.23||1||1||1|
|2.3.1 - Forest Ecosystem Science and Application||2.3.1 - Forest Ecosystem Science and Application||CFS||15,000||0||15,605,942||1.97|
|2.3.2 - Groundwater Geoscience||2.3.2 - Groundwater Geoscience||LMS||0||0||3,233,122||0.41|
|2.3.3 - Environmental Studies and Assessments||2.3.3 - Environmental Studies and Assessments||LMS||R||20-21 Q1||21-22 Q2||0||0||7,287,409||0.92||4||50||2|
|2.3.4 - Geospatial Information for Responsible Natural Resource Development||2.3.4 - Geospatial Information for Responsible Natural Resource Development||SPRS||0||0||1,311,150||0.17|
|3.1.1 - Explosives Safety and Security||3.1.1 - Explosives Safety and Security||LMS||R||18-19 Q3||20-21 Q1||0||0||4,524,427||0.57||3||25||6||30||1|
|3.1.1 - Explosives Safety and Security||Single Window Initiative||LMS||TB||CBSA||18-19 Q1||18-19 Q4||0||0||760,000||0.10||2|
|3.1.2 - Materials and Certification for Safety and Security||3.1.2 - Materials and Certification for Safety and Security||LMS||0||0||1,860,386||0.23|
|3.1.3 - Forest Disturbances Science and Application||3.1.3 - Forest Disturbances Science and Application||CFS||R||15-16 Q3||17-18 Q2||43,000||0.01||40,735,113||5.14||1|
|3.1.4 - Climate Change Adaptation||3.1.4 - Climate Change Adaptation||LMS||TB||ECCC||19-20 Q3||21-22 Q4||1,000,000||0.31||7,042,592||0.89||1||2||1|
|3.1.5 - Geohazards and Public Safety||3.1.5 - Geohazards and Public Safety||LMS||TB||20-21 Q3||21-22 Q4||35,000||0.01||15,495,561||1.95||5||7|
|3.2.1 - Essential Geographic Information||3.2.1 - Essential Geographic Information||LMS, SPRS||R||15-16 Q2||17-18 Q2||10,000||0||13,226,861||1.67||1|
|3.2.1 - Essential Geographic Information||Geomatics - Remote Sensing and Mapping||CFS||R||18-19 Q3||19-20 Q4||0||0||13,226,861||1.67||4||10||30|
|3.2.2 - Canada's Legal Boundaries||3.2.2 - Canada's Legal Boundaries
Joint audit and evaluation
|LMS||R||18-19 Q3||19-20 Q4||0||0||12,851,311||1.62||4||8||30|
|3.2.3 - Polar Continental Shelf Logistics Support||3.2.3 - Polar Continental Shelf Logistics Support||LMS||0||0||8,569,697||1.08|
|3.2.4 UN Convention on the Law of the Sea||Extended Continental Shelf||LMS||TB||GAC||17-18 Q2||17-18 Q4||0||0||1,275,447||0.16||2|
|4.1.1 Management and Oversight||Federal Contaminated Sites Action Plan||CMSS||TB||ECCC||16-17 Q4||18-19 Q2||0||0||3,030,000||0.383||1||1|
|Other||Nuclear Legacy Liabilities Program4||ES||TB||15-16 Q3||17-18 Q2||--||--||--||--||.5|
|Other||Class Grants and Contributons5||All||TB||15-16 Q3||17-18 Q2||--||--||--||--||.5|
|Other||Neutral Assessment||18-19 Q1||18-19 Q4||2||25|
1 PM or person month is a unit of measurement reflecting level of effort. One person month is equal to twenty 7.5 hours days.
2 The latter three years of the plan have purposefully been under-scheduled in order to accommodate the large number of evaluation commitments related to 2017-18 Treasury Board submissions.
3 The proportion of direct program spending accounted for by the FCSAP program is not included in the total, as it is aligned to an internal services sub-program. Total values reflect total program spending excluding internal services.
4 Program has sunset. As such, there is no 2017-18 grants and contribution (G&C) or direct program spending (DPS) associated with this program
5 The value of the class grants and contributions program spending is listed next to each of the sub-programs that make use of these terms and conditions.
CFS = Canadian Forest Service
CMSS = Corporate
ES = Energy Sector
IETS = Innovation and Energy Technology Sector
LMS = Lands and Minerals Sector
SPRS = Strategic Policy and Results Sector
G&C = FAA 42(1) requirement
R = Risk-based
TB = Treasury Board Commitment
CBSA = Canada Border Services Agency
ECCC = Environment and Climate Change Canada
ESDC = Employment and Social Development Canada
GAC = Global Affairs Canada
TC = Transport Canada
= Active in 2017-18
= Active in 2018-19 or afterward
= Not currently scheduled for evaluation
- Date modified: