Evaluation of Forest Ecosystems Science and Application Program Sub Activity

Table of Contents

• Executive Summary
• Acronyms
• 1.0 Introduction
• 2.0 Program Background and Profile
  • 2.1 Context
  • 2.2 Mandate and Stakeholders
  • 2.3 Governance and Administration
  • 2.4 Activities, Delivery and Organization
    • 2.4.1 Forest Biodiversity
    • 2.4.2 Forest Carbon Research, Reporting and Policy Advice
    • 2.4.3 Assessing and Understanding Ecosystem Productivity and Dynamics in Support of Sustainable Forest Management
    • 2.4.4 Global Leadership in the Development of the International Model Forest Network (IMFN)
    • 2.4.5 National Forest Information and Assessment (NFIA)
    • 2.4.6 Reclamation of Disturbed Forest Landscapes
    • 2.4.7 Distribution of components by project area
  • 2.5 Funding and Resources
  • 2.6 FESA Logic Model
• 3.0 Evaluation Objectives, Scope, and Methodology
  • 3.1 Objectives and Scope
  • 3.2 Methods
  • 3.3 Evaluation Challenges, Limitations and Mitigation Strategies
    • 3.3.1 Challenges
    • 3.3.2 Data limitations
• 4.0 Evaluation Findings
  • 4.1 Relevance
    • 4.1.1 Continued Need for the Program (Issue 1)
    • 4.1.2 Alignment with Government Priorities (Issue 2)
    • 4.1.3 Alignment with Federal Roles and Responsibilities (Issue 3)
  • 4.2 Performance
    • 4.2.1 Achievement of Expected Outcomes (Issue 4)
    • 4.2.2 Demonstration of Efficiency and Economy (Issue 5)
• 5.0 Conclusions and Recommendations

Tables

• Table 1 FESA Program Sub-Activity stakeholders
• Table 2 Activities and outputs expected for the Biodiversity project area
• Table 3 Activities and outputs expected for the Carbon project area
• Table 4 Activities and outputs expected for the Productivity & Dynamics project area
• Table 5 Activities and outputs expected for the IMFN project area
• Table 6 Activities and outputs expected for the Forest Assessment project area
• Table 7 Activities and outputs expected for the Land Reclamation project area
• Table 8 Number of components by project area, 2007-08 to 2011-12
• Table 9 Summary of internal and external expenditures, by FESA project area and overall, 2007-08 to 2011-12 ($ millions)
• Table 10 FESA Financial Expenditures by type and by project area, 2007-08 to 2011-12 ($ millions)
• Table 11 Overview of evaluation data collection methods
• Table 12 Peer-reviewed scientific output in Forestry, 2003–2011
• Table 13 CFS output by FESA project area and document type, 2003–2012
• Table 14 CFS publication output by FESA project area and CFS research centre, 2003–2012
• Table 15 Result of webpage usage study for eight selected peer-reviewed papers
• Table 16 Selected CFS grey literature documents, produced between 2003 and 2012, with most Google hits per FESA project area
• Table 17 Webmetrics results for selected FESA non-publication outputs
• Table 18 Result of webpage usage study for eight selected non-publication items

Figures

• Figure 1 FESA Program Sub-Activity Logic Model
• Figure 2 Specialization(*) patterns of Canadian sectors within Forestry across FESA project areas, (A) 2003-2006 and (B) 2008-2011
• Figure 3 Main types of outputs identified by FESA component leads
• Figure 4 Primary and secondary target audiences for FESA outputs
• Figure 5 CFS publication output by FESA project area and year, 2003–2012
• Figure 6 Extent to which FESA components contribute to understanding forest ecosystem responses to natural factors and human activity
• Figure 7 Extent to which FESA components contribute to identifying forest ecosystem risks and science priorities
• Figure 8 Extent to which target audiences use FESA outputs to manage forest ecosystems, and forecast and manage forest ecosystem risks
• Figure 9 Extent to which target audiences within governments use FESA outputs to inform decisions that improve forest ecosystem management policies and practices
• Figure 10 Extent to which non-governmental target audiences use FESA outputs to inform decisions that improve forest ecosystem management policies and practices
• Figure 11 Factors facilitating or inhibiting achievement of FESA outcomes
• Figure 12 Trends in internal resources by project area, 2007-08 to 2011-12
• Figure 13 Trends in internal resources allocated as a percentage of funding requested, by project area, 2007-08 to 2011-12
• Figure 14 Trends in external financial resources by project area, 2007-08 to 2011-12

Acronyms

Executive Summary

Introduction

This is an evaluation of the Forest Ecosystems Science and Application (FESA) Program sub-activity for the period 2007-08 to 2011-12. FESA was established with the objective to increase scientific knowledge on forest ecosystems and support stakeholders in their sustainable forest management (SFM) policies and practices. As part of Natural Resources Canada (NRCan), FESA conducts research, national assessments and monitoring to develop, synthesize and integrate scientific knowledge. This knowledge is used by the appropriate government jurisdictions, industry, and other stakeholders to develop forest management practices and policies, and by NRCan and other federal government departments, to meet international reporting obligations, from Canada’s negotiating positions on international environmental issues related to forests, and counter misconceptions of Canada’s forest practices. This work is conducted by the Canadian Forest Service (CFS) within NRCan.

During the evaluation period, this Program Sub-Activity consisted of the following six project areas: 1. Forest Biodiversity; 2. Forest Carbon Research, Reporting and Policy Advice; 3. Assessing and Understanding Ecosystem Productivity and Dynamics in Support of Sustainable Forest Management; 4. Global Leadership in Development of the International Model Forest Network (IMFN); 5. National Forest Information and Assessment (NFIA); and 6. Land Reclamation.

Note that FESA also encompasses a regional delivery model, with work carried out in the National Capital Region, as well as in five different forestry centres in Canada. Expenditures for delivery for the FESA Sub-Activity totaled $101.5 million from 2007-08 to 2011-12, including $84.4 of NRCan expenditures, $5.7 million in external financial resources (actual) and $11.4 million in-kind support (planned).

Evaluation Scope, Objectives and Methods

This evaluation was mandated by NRCan in accordance with the Federal Accountability Act, the 2009 Treasury Board Policy on Evaluation and NRCan’s Strategic Evaluation Division’s evaluation policies. The primary objective of the evaluation was to assess issues relating to the relevance and performance of the FESA Program Sub-Activity and provide recommendations as necessary.

All six project areas were covered by the evaluation, with the exception of the African Model Forest Initiative, a Contribution program within the IMFN project area, which will be assessed as a separate evaluation. The FESA evaluation followed a theory-based approach, in which emphasis is placed on the process by which the Program arrives at its expected results.

The evaluation was designed to draw on various sources of data to ensure that the combined lines of evidence resulted in an in-depth and comprehensive analysis. The five methods used to collect and analyze evidence were: 1) a document file and data review; 2) a review of CFS ProMIS planning database; 3) stakeholder interviews and focus groups; 4) an online survey of project area component leads; and 5) bibliometrics and webmetrics.

Evaluation Findings

Relevance

Continued need for program: FESA remains highly relevant, addressing several ongoing needs. FESA provides the unique and long-standing expertise and specialization in forest ecosystem science that are particularly relevant to address environmental issues and to support forestry sector competitiveness. Ongoing needs for each of the six FESA project areas were also identified by the evaluation. However, CFS should clearly define its niche with respect to the Land Reclamation project area.

Alignment with federal roles and responsibilities: The evaluation found that FESA plays a legitimate and key role, gathering and reporting national information used by a wide range of target audiences, as well as building a long-term perspective. CFS provides unique scientific expertise and research and complements those outputs from provinces/territories and academia. Furthermore, the federal government is mandated to represent Canada’s global interests and address international obligations relating to forestry.

The alignment of the IMFN and Biodiversity project areas to FESA requires further clarification. The IMFN as a platform to import forest management knowledge and experiences is relevant to more than one Sub-Activity Program, including FESA. Therefore, there may be an opportunity to position IMFN as a platform for disseminating horizontal CFS scientific research on sustainable forest management. Regarding Biodiversity, its integration into a new project area (Ecosystems Integrity) has helped clarify and focus the scope of research to be conducted; however, this has resulted in biodiversity expertise that is perceived by some NRCan interviewees to not be fully integrated within the new project area. There is an opportunity to further assess whether the biodiversity research or expertise developed over the last five years has been adequately integrated into the Ecosystems Integrity project area.

Alignment with government priorities: FESA was consistent with both federal government priorities and NRCan strategic outcomes throughout the evaluation period. Federal priorities over the evaluation period consistently displayed a commitment to developing the forest industry. FESA was also aligned with NRCan goals to remain environmentally responsible as outlined in the Program Alignment Architecture (PAA).

Performance

Achievement of expected outcomes: The FESA project areas have been generally successful in producing high quality outputs, which have been accessed and used by a wide range of target audiences. The work of CFS is regarded as credible and science-based such that provinces/territories often use FESA information or expertise to fill their own capacity gaps. FESA has also realized immediate outcomes and contributed to some intermediate level outcomes. FESA activities have contributed to an increased understanding of how ecosystems respond to natural factors and human activity, have supported domestic policy making and improved forest management practices, and have contributed to Canada meeting its national and international obligations. While the evaluation could not clearly determine the extent to which FESA information is used to forecast and manage ecosystem risks, the existing evidence suggests that FESA does moderately contribute to some risk management activities.

The Sub-Activity’s stakeholders were generally satisfied with the accessibility of FESA outputs; however, some opportunities for improvement were identified with respect to the communication and usability of these outputs and the need for more direct interactions with stakeholders.

Importantly, during the evaluation period, CFS has increasingly oriented its science towards policy priorities within available resources. This is likely to have an effect on capacity, expertise, and services historically made available to traditional end-users, as well as regional partners. The evaluation finds that to date, these changes have been effectively implemented internally, but not well-communicated to users.

Demonstration of efficiency and economy: Collaborations, partnerships and close working relationships were found to greatly facilitate the achievement of outcomes and enhance external leveraging. Support from external resources (funding and in-kind) often made up the difference between the requested and allocated internal resources in several project areas. This is significant considering that internal resources have declined overall during the evaluation period.

Considering the synergistic working relationships between CFS and its traditional users, there is a need for FESA to communicate to its stakeholders any changes in its services, including those due to an increasing focus on science-policy integration.

Declining resources also posed a challenge to supporting access and use of FESA information by external stakeholders. Lack of public science dissemination and/or communication of new tools or outputs meant that users were not always aware of FESA achievements or that FESA outputs were not in the best format to be efficiently used.

Although there are management procedures in place to support the achievement of FESA outcomes in an economical and efficient manner, there is currently no systematic performance data collection or progress reporting process. Given that this is a sub-activity of national scope with diverse project areas and target audiences, improving the priority-setting process and enhancing the ProMIS system could allow for better performance tracking and hence more informed decision-making.

Recommendations and Management Response

The following Table presents six recommendations, based on the findings, and the management responses.

Recommendations	Management Response/Action Plans	Responsible Official/Sector (Target Date)
Recommendation 1: It is recommended that NRCan further examine the relevance of FESA’s role in the International Model Forest Network and Land Reclamation. CFS should explore opportunities to position IMFN as a platform for disseminating CFS scientific research and knowledge on sustainable forest management. CFS should also clearly identify the objectives and key research questions that Land Reclamation needs to address.	Agreed. Actions to address the recommendation: (i) The CFS agrees to further capitalize on IMFN as a FESA project aiming at communicating CFS science and disseminating knowledge on sustainable forest management policy and practices. In particular, all opportunities across CFS programs for using IMFN to highlight Canadian best practices in forest management in support of our environmental reputation will be explored. (ii) The CFS is currently assessing the strategic drivers for the Land Reclamation Project as it relates to FESA and Canada’s boreal zone. The CFS has also initiated work within the broader context of NRCan with external stakeholders to identify data and knowledge gaps related to Land Reclamation. As a key priority for the GoC and NRCan, CFS will evaluate and re-focus the investment in the Project commensurate with federal priorities, as part of its on-going resource allocation exercise.	ADM/CFS March 2015
Recommendation 2: Given the partial integration of the Biodiversity project area into the Ecosystem Integrity project area, it is recommended that NRCan further examine any risks associated with the reallocation and/or reduction of biodiversity expertise for CFS and the federal government.	Agreed. Actions to address the recommendation: (i) Within the context of the new Ecosystem Integrity Project, CFS will undertake a risk assessment to determine if the integration of the Biodiversity activities into the Project would compromise its ability to provide expertise on biodiversity to CFS and GoC.	ADM/CFS March 2015
Recommendation 3: Given the evolution of CFS science towards meeting federal policy priorities, it is recommended that NRCan clearly communicate its current and longer term plans to provinces (and other key users) by specifying what changes in expertise and outputs traditional users can expect. CFS should also consider enhancing dialogue between FESA scientists and CFS/NRCan policy staff to help clarify communication plans for traditional and new users.	Agreed. Actions to address the recommendation: (i) CFS has shared with Provinces and all staff its Strategic Framework that aligns science effort with policy needs to support federal priorities. (ii) In addition, CFS will continue to communicate with provinces and other stakeholders regarding our science outputs through the large number of boards and committees on which it sits. The active participation of scientists and policy analysts to FESA activities has been identified to strengthen the communication and understanding of the policy needs of NRCan/CFS. (iii) The CFS will ensure that the science-policy integration approach supports the NRCan strategic outcomes and its sector risk profile (that is updated at least annually). Thus, CFS will revise its strategic plan for the FESA starting in FY 2014-2015 to reflect its commitment in this regard.	ADM/CFS March 2015
Recommendation 4: Within the context of limited resources, it is recommended that NRCan enhance opportunities for its scientists to interact more frequently with target audiences, to ensure the effectiveness of the Sub-Activity and enable FESA to secure more external resources (financial and in-kind).	Agreed. Actions to address the recommendation: (i) Through its scientists, CFS will continue to emphasize knowledge exchange activities at the level of regional, national and international fora without compromising the TBS Travel Directive. (ii) CFS will explore and use innovative communication tools (webinars, web-surveys etc.) and capitalize on its strong partnerships to communicate CFS science to target audiences.	ADM/CFS March 2015
Recommendation 5: Within the context of limited resources, it is recommended that NRCan enhance the public dissemination of the science produced by the Sub-Activity.	Agreed. Actions to address the recommendation: i) CFS will undertake outreach and communication activities to disseminate scientific information on FESA science by: a) highlighting Canadian science for forest stewardship and forest sustainability by using science communications aimed at the public; and b) Incorporating knowledge exchange and other communication activities into the planning of FESA science projects.	ADM/CFS March 2015
Recommendation 6: It is recommended that NRCan continue to improve the tracking and reporting of performance and financial data to ensure that reliable information is used to manage its activities.	Agreed. CFS will take the following measures in this regard: (1) The CFS is currently working towards refining performance indicators for tracking our science performance and for linking these to our investments. This will also help clarify the targeted end-users. (2) A continual improvement in reporting information on finances, activities, outputs and outcomes will be available for subsequent years where any NRCan financial systems will enable tracking of resources for FESA activities and this expenditure information will be available shortly after FY-end.	ADM/CFS March 2015

1.0 Introduction

This report presents the findings, conclusions and recommendations for the evaluation of Forest Ecosystems Science and Application (FESA) Program Sub-Activity.

The FESA Program Sub-Activity was established with the objective to increase scientific knowledge on forest ecosystems and support stakeholders in their sustainable forest management (SFM) policies and practices. This Program Sub-Activity is responsible for conducting research, national assessments and monitoring to develop, synthesize and integrate scientific knowledge. At the time of the evaluation, the FESA activities represented sub-activity 2.2.2 in the NRCan Program Activity Architecture (PAA) under Strategic Outcome #2, Environmental Responsibility.

The evaluation examined the issues of relevance and performance and covered the period from fiscal year 2007-08 to 2011-12. It included all six project areas under the FESA Sub-Activity with the exception of the African Model Forest Initiative (within the IMFN project area) that will be evaluated separately. The evaluation was conducted between November 2012 and August 2013 in accordance with the 2009 Treasury Board Policy on Evaluation and the Departmental Strategic Evaluation Plan (2012-13 to 2016-17).

The evaluation design and findings were informed by feedback from an Evaluation Advisory Committee (EAC) composed of management representatives from each project area of the Sub-Activity. Multiple lines of evidence were integrated, resulting in preliminary findings which were then presented to the Evaluation Advisory Committee (EAC) for validation.

2.0 Program Background and Profile

2.1 Context

Canada has almost 400 million hectares of forest, wooded or tree-covered land, representing about 10% of the world’s forest cover and 30% of the world’s boreal forest. Forests are important to the Canadian economy, with forest-related industries contributing about 1.9% to Canada’s gross domestic product in 2011^{Footnote 1} , as well as 222,000 direct jobs.^{Footnote 2}

In addition to these direct economic values, forests worldwide are recognized as being important locations for biodiversity as well as serving ecological functions, such as regulating the hydrological cycle, protecting watersheds and storing genetic information.^{Footnote 3} There are new demands from both the Canadian public and environmental organizations for recognition of the entire range of environmental goods and services that forests supply beyond resource extraction.^{Footnote 4} At the same time, international environmental organizations are questioning the health of Canada’s forest ecosystems and the Canadian commitment to environmental stewardship.^{Footnote 5} In addition, studies on Canadian forests forecast significant changes in forest ecosystems over the coming decades, due to climate change.^{Footnote 6}

The impacts of climate-induced changes, natural influences (i.e., disease), forest harvesting and land-use changes point to a need to monitor and assess the health of Canada’s forest ecosystems to support their sustainable development.^{Footnote 7} The FESA Program Sub-Activity is designed to produce scientific knowledge of Canada’s forest ecosystems that the public, government, industry and environmental organizations need for the responsible forest stewardship.^{Footnote 8}

2.2 Mandate and Stakeholders

The objective of the FESA Program Sub-Activity is to increase scientific knowledge on forest ecosystems and support stakeholders in their SFM policies and practices. As part of this Program Sub-Activity, NRCan conducts research, national assessments and monitoring to develop, synthesize and integrate scientific knowledge. This knowledge is used by the appropriate government jurisdictions, industry, and other stakeholders to develop forest management practices and policies, and by NRCan and other federal government departments, to meet international reporting obligations, from Canada’s negotiating positions on international environmental issues related to forests, and counter misconceptions of Canada’s forest practices.^{Footnote 9} This work is conducted by the Canadian Forest Service (CFS) within NRCan. While it is important for CFS to produce scientific outputs that meet their stakeholders’ needs, the organization has recently made a shift towards better science-policy integration, in order to ensure that research is aligned with federal policy priorities in the area of forest ecosystems.

Activities of the FESA Program Sub-Activity are primarily in the following areas:^{Footnote 10}

• Developing and maintaining national information on Canada’s forest ecosystems: to assess the state of the ecosystems and the indicators of ecosystem health, to assist in meeting Canada’s international reporting obligations, and to strengthen the understanding of Canada’s forest resource development standards and practices.
• Understanding ecosystem function and how it is impacted by resource development: to delineate risks to forest ecosystems posed by natural resource development and other anthropogenic or natural processes.
• Providing improved tools and practices needed to strengthen the integration of responsible environmental stewardship and forest resource development: to mitigate the anthropogenic risks.
• Exchanging knowledge and best practices domestically and internationally: to promote the responsible development of forest resources in Canada and globally.

As of 2011-12, this Program Sub-Activity consisted of the following six project areas. Note that Forest Biodiversity and Productivity & Dynamics merged into one project area, Ecosystems Integrity, in 2012-13 which is outside the evaluation reference period.

1. Forest Biodiversity (Biodiversity), to determine the impacts of forest resource development and natural processes on biodiversity and develop a landscape-level risk management framework to aid in prioritizing NRCan-CFS forest research. A modest level of effort is focused on the impacts of forest management on the habitats of endangered species (e.g., woodland caribou) which are sometimes characterized as indicator species of biodiversity.
2. Forest Carbon Research, Reporting and Policy Advice (Carbon) assesses forest-related carbon stock changes and greenhouse gas (GHG) emissions as an input into Canada’s reporting on GHG emissions – including to meet its annual obligations under the United Nations Framework Convention on Climate Change (UNFCCC) and (formerly) under the Kyoto Protocol. In addition, it supports integration of forest carbon and GHG considerations into federal policy development and in improved forest management by providing science- and economics-based assessments of how forests can contribute to climate change mitigation. In support of this integration, it also transfers carbon modeling tools to stakeholders.
3. Assessing and Understanding Ecosystem Productivity and Dynamics in Support of Sustainable Forest Management (Productivity & Dynamics), to assess the impacts of forest resource development on productivity, paying attention to the multiple factors at various scales in time and space (e.g., global environmental change, forest practices, wildfires, pests and other agents of tree mortality). This work aims to ensure that forest ecosystems remain resilient and productive. This work also improves the understanding of complex ecosystem dynamics related to forest productivity.
4. Global Leadership in Development of the International Model Forest Network (IMFN), intended to be used as a platform to share Canadian best practices, knowledge of, and tools to promote responsible forest development globally; and to import knowledge and experiences gained elsewhere to aid in developing solutions to forest management challenges in Canada.
5. National Forest Information and Assessment (Forest Assessment or NFIA) delivers an annual report to Parliament on the state of Canada’s forests (i.e., the annual State of the Forests Report), supports Canada’s reporting obligations under international agreements, and helps support market access issues. This work contributes to the development of products that provide information and advice on national forests (e.g., the National Forest Inventory) as well as contributing to the broader integration of systems for monitoring activities and research across disciplines.
6. Land Reclamation research on reclaiming mined oil sands sites so that healthy, productive forests are restored. This also considers other ecosystem restoration activities over the longer term.

Key stakeholders for the six FESA areas are outlined in Table 1. Some of these stakeholders are considered to be clients, partners or users; however, roles may overlap (e.g., a client or partner could also be an information user).

Table 1 FESA Program Sub-Activity stakeholders

Stakeholders	Project areas
	Bio-diversity	Carbon	Productivity & Dynamics	IMFN	Forest assessment	Land Reclamation
Other CFS/NRCan programs	•	•		•4		•
Other federal departments	•1	•2	•	•5	•7	•8
Provinces/territories	•	•3	•		•	•9
Hydro organizations	•
Universities	•	•	•		•	•
Forest industry and practitioners		•	•		•	•
External funding partners				•6
International organizations		•		•
Non-governmental organizations		•			•	•
Canadian public					•
Oil industry						•
Oil-related associations						•

Notes: 1 - EC, DFAIT; 2 - EC, AAFC; 3 – via the National Forest Sinks Committee, which engages in policy-related analysis and facilitates provision of data needed by CFS to meet Canada’s forest carbon and GHG reporting commitments; 4 – the 60 sites on the network; 5 – CIDA, DFAIT; 6 – governments of other countries; 7 – CSA, AAFC; 8 – DFO, EC; 9 – primarily Alberta.

Source: Adapted from the Evaluation Assessment Report prepared by NRCan’s Strategic Evaluation Division (SED), citing NRCan ProMIS 2011-12.

2.3 Governance and Administration

The FESA Program Sub-Activity operates under the authority of the Natural Resources Act (1994) and the Forestry Act (1985). As per the former, the Minister shall, “seek to enhance the responsible development and use of Canada’s natural resources and the competitiveness of Canada’s natural resource products”.^{Footnote 11} Furthermore, as a result of the 2010 Federal Sustainable Development Strategy, the Minister of Natural Resources is responsible for, “advancing knowledge and communication” specifically by generating and disseminating “scientific knowledge related to forest ecosystems”.^{Footnote 12}

The FESA Program Sub-Activity project areas report through the Assistant Deputy Minister of NRCan’s Canadian Forest Service (CFS). Each of the project areas within the Program Sub-Activity has a project coordinator. With the exception of the International Model Forest Network, each project coordinator is supported by peer directors such that the team responsible for a project area consists of a member from headquarters and members of regional research centres across Canada^{Footnote 13} . The project coordinators report through two Directors General (DG Laurentian Forestry Centre and DG Northern Forestry Centre), who act as co-leads for this Program Sub-Activity.

2.4 Activities, Delivery and Organization

The FESA Program Sub-Activity encompasses a regional delivery model, with work carried out in the National Capital Region, as well as in five forestry research centres:

• Atlantic Forestry Centre (New Brunswick and Newfoundland)
• Great Lakes Forestry Centre (Ontario)
• Laurentian Forestry Centre (Quebec)
• Northern Forestry Centre (Alberta)
• Pacific Forestry Centre (British Columbia)

The six project areas of the FESA Program Sub-Activity differ in their objectives, activities and outputs, number of components, stakeholders, etc. This section presents a brief description of the primary activities, and expected outputs for each.^{Footnote 14}

2.4.1 Forest Biodiversity

Description

The Forest Biodiversity (Biodiversity) project area operated from 2007-08 through 2011-12. The objective of this project area was to conduct nationally-relevant forest biodiversity science to support the provision of knowledge and advice to the forest sector that would strengthen Canada’s ability to demonstrate to markets the science-based foundation of Canada’s forest policy.^{Footnote 15} The context for this project area was the recognition that the pressures on the forest sector’s environmental reputation are largely related to biodiversity.^{Footnote 16}

Activities and Outputs

Four primary activities were identified and aligned with five principal outputs as outlined in Table 2.

Table 2 Activities and outputs expected for the Biodiversity project area

Activities

Outputs

Determine the impacts of forest resource development on natural processes and biodiversity Develop a landscape-level risk management framework to help mitigate impacts Delineate the impacts of forest management on habitat of endangered species Exchange new knowledge with policy makers and resource managers

A synthesis of science information on Canada’s boreal forest A landscape-level risk management framework for understanding impacts of forest management on biodiversity and options for mitigation Assessments of the effects of forest management on the critical habitat of woodland caribou and pine marten in the boreal forest Analysis of emerging issues in forest biodiversity Science advice, synthesis and communication of research findings

2.4.2 Forest Carbon Research, Reporting and Policy Advice

Description

Since 2007-08, the Forest Carbon Research, Reporting and Policy Advice (Carbon) project area has developed scientific knowledge, models, reports and policy advice to inform/influence decision-making on the management of forest-related carbon and greenhouse gas emissions.^{Footnote 17}

The objectives of this project area are to:^{Footnote 18}

• enhance and integrate scientific knowledge about the determinants of forest carbon and greenhouse gas dynamics across various scales and the impacts of management;
• maintain and develop the National Forest Carbon Monitoring, Accounting and Reporting System to support policy and annual national/international reporting, including under the UNFCCC; and
• support the integration of forest carbon and greenhouse gas considerations in improved forest management by providing science- and economics-based assessment of how forests can contribute to climate change mitigation, and by transferring forest carbon/greenhouse gas information and assessment tools to stakeholders.

Activities and Outputs

Three primary activities and four outputs expected for the project are presented in Table 3.

Table 3 Activities and outputs expected for the Carbon project area

Activities	Outputs
Investigating and delineating carbon dynamics and the impacts of human activity on forest carbon Quantifying carbon removals and GHG emissions related to forests for input into Canada’s greenhouse gas emissions report Generating new knowledge to be exchanged with policy makers and resource managers	Knowledge of forest carbon dynamics Estimates of the impacts of forestry practices on carbon Annual estimates and reporting on the forest sector greenhouse gas budget Assessments and tools for addressing forest carbon policy issues

2.4.3 Assessing and Understanding Ecosystem Productivity and Dynamics in Support of Sustainable Forest Management

Description

The Assessing and Understanding Ecosystem Productivity and Dynamics in Support of Sustainable Forest Management (Productivity & Dynamics) project area existed from 2007-08 through 2011-12. It was intended to generate and integrate scientific knowledge, models and decision-support tools and policy advice for decision makers as well as to maintain market access.^{Footnote 19} Areas of focus were: understanding the impact of forest practices and climate change on ecosystem function; tracking and reporting on trends in Canada’s forest productivity, and providing baseline information for decision making regarding climate change mitigation.^{Footnote 20}

The objectives of this project area were to:^{Footnote 21}

• understand ecosystem functions and how they are impacted by forest practices and global change;
• track and report on the trends in Canada’s forest productivity over time; and
• provide baseline information for sound decision making, notably for climate change mitigation.

Activities and Outputs

Three primary activities and four outputs expected for the project area are presented in Table 4.

Table 4 Activities and outputs expected for the Productivity & Dynamics project area

Activities	Outputs
Assess the impacts of resource development on forest productivity Determine the ecosystem dynamics of productivity levels to support development of lower impact forestry practices Generate new knowledge to exchange with policy makers and resource managers	National mapping of forest landscape change influencing forest dynamics and productivity Increased understanding of forest ecosystem processes as a function of the impacts of multiple forest disturbances Increased understanding of the impacts of forest practices and global change on forest ecosystem productivity Ecosystem-based management framework for Canada’s forests

2.4.4 Global Leadership in the Development of the International Model Forest Network (IMFN)

Description

The Global Leadership in the Development of the International Model Forest Network (IMFN) project area is a global community of practice in which members work towards the sustainable management of forest landscapes that has existed since 1992.^{Footnote 22} At the 1992 United Nations Conference on Environment and Development in Rio de Janeiro, the Government of Canada announced that it would “be the international counterpart to Canada’s domestic model forest program”.^{Footnote 23} The purpose of this was to “stimulate the field-level application of new concepts and ideas in SFM in forest ecosystems throughout the world and to create opportunities to demonstrate and share these experiences”.^{Footnote 24}

Within this project area, there are essentially 5 components:^{Footnote 25}

• Model Forests: defined as a geographic area and a specific approach to SFM. They are large-scale, long-term experiments for managing forested landscapes in a manner that adheres to six common principles while promoting and improving ecological, economic, and social sustainability. The six common principles are: broad, inclusive partnerships; landscape scale; commitment to sustainability; transparent, accountable governance; program of activities reflective of partner values and interests; and knowledge sharing, capacity building and networking.
• International Model Forest Network: as of March 20012 the IMFN consisted of over 60 model forests (14 Canadian, and 45 either established or under development outside of Canada) and had three main objectives: (1) to foster international cooperation and exchange of ideas relating to the working concept of SFM; (2) to support international cooperation in critical aspects of forest science and social science that underlie the search for new models of forest management; and (3) to support ongoing international discussions on the criteria and principles of sustainable development.
• IMFN Secretariat: currently housed within NRCan CFS^{Footnote 26} and is responsible for the implementation of the IMFN program. Its role is to facilitate the development of a network of model forests that is dedicated to managing the world’s forest-based landscapes in a sustainable manner. Specifically, the IMFN Secretariat provides centralized coordination of day-to-day support and development services to the Network, works to strengthen and expand the Network, and supports new and existing model forests when there is no regional network support in place at the site-level.
• Regional Model Forest Networks: through the IMFN, Secretariat regional networks have been established in Latin America, Africa, Europe, Asia and the Mediterranean to more effectively define, articulate and manage a regional program of SFM. The Canadian Model Forest Network is a member of the IMFN and is treated as a regional network of the IMFN. These regional networks reflect the unique priorities, strengths and opportunities in that specific region. They also facilitate regional communication and knowledge exchange, capacity building and funding opportunities for existing and potential model forests. The network is therefore very non-hierarchical, with significant autonomy and responsibility vested in regions with the overall effort functioning in a voluntary (non-legally binding) manner.
• Model Forests in Africa: as of July 2008, there were two model forests in Africa (both in Cameroon), the development of which were started by the IMFN Secretariat in 2003. In March 2008, North Africa, Algeria, Morocco and Tunisia indicated an interest in joining a regional network and the IMFN. By the end of 2012, a total of 8 Model Forests were in various stages of development in Africa. These Model Forests are supported by local authorities within each country’s government and by a national coordinator for the African Model Forest Network.^{Footnote 27} The objective of the African Model Forest Initiative (AMFI) is to “improve the conservation and sustainable management of forest resources in the Congo Basin and the Mediterranean regions of Africa”. The Africa Model Forest Network is unique within the IMFN, having been funded largely through a $15 million contribution from Canada.

Activities and Outputs

Three primary activities and four outputs expected for the project area are presented in Table 5.

 

Table 5 Activities and outputs expected for the IMFN project area

Activities

Outputs

Sharing Canadian knowledge of and tools for practicing sustainable forest management Promoting responsible forest resource development internationally Leadership in development of the IMFN intended to be used as a platform to import knowledge and experiences gained elsewhere to aid in developing solutions to forest management challenges in Canada (such as environmental problems, best practices around aboriginal engagement and stakeholder participation) Footnote 28

Coordinated research on key global forest issues and trends through network initiatives Enhanced links with strategic institutional partners and countries Enhanced knowledge management capacity throughout the IMFN Enhanced local, regional and global governance for sustainable resource development

2.4.5 National Forest Information and Assessment (NFIA)

Description

The National Forest Information and Assessment (Forest Assessment or NFIA) project area has existed since 2007-08 to provide fundamental information on the forest resources and ecosystems.^{Footnote 29} This information is required for Canada to meet its science, policy, program and reporting commitments.^{Footnote 30}

The objectives of this project area are to:^{Footnote 31}

• track changes in Canada’s forest resources, conduct analysis and forecasting that will support decision-making related to policy, investment and trade; and
• provide information to support regional, national and international business objectives and science initiatives of CFS.

Activities and Outputs

Three primary activities and four outputs expected for the project area are presented in Table 6.

Table 6 Activities and outputs expected for the Forest Assessment project area

Activities	Outputs
Conducting national assessments and generating information on the state of Canada’s forests Develop and maintain databases needed for the assessments Strengthening national forest information tools for information gathering	Data on Canada’s forests Methods and process for improved information and assessment of Canada’s forests Reporting on Canada’s forests National and international collaboration

2.4.6 Reclamation of Disturbed Forest Landscapes

Description

The development of oil sands in Alberta has had significant environmental impacts on forested landscapes and watersheds.^{Footnote 32} The Reclamation of Disturbed Forest Landscapes (Land Reclamation) project area was created in 2011-12 to conduct research on reclaiming mined oil sands sites to enable the restoration of healthy, productive forests. This research is being conducted in collaboration with NRCan’s Innovation and Energy Technology Sector.^{Footnote 33}

The objectives of this project area are to:^{Footnote 34}

• assess the current state of knowledge on reclamation practices designed to ensure that re-established ecosystems provide essential ecosystem goods and services;
• identify knowledge gaps;
• generate knowledge to support effective mitigation and reclamation practices; and
• develop tools to enable policy and decision-making.

Activities and Outputs

Two primary activities and five outputs expected for the project area are presented in Table 7.

Table 7 Activities and outputs expected for the Land Reclamation project area

Activities

Outputs

Conducting research on reclaiming mined oil sands sites Exchanging new knowledge with policy makers and resource managersFootnote 35

Reviews and synthesis of the current state of knowledge (socio-economic, scientific and cultural historical) on landscapes reclamation practices following mining disturbances in Canada and identify key scientific gaps that could be addressed by CFS. Characterization and quantification of ecosystem services generally provided on landscapes that are destined for oil sands resource extraction to inform future reclamation efforts Assessments of the current and developing reclamation activities and their potential to address the cumulative impacts of oil sands development for the reestablishment of sustainable ecosystem processes outlined in output 2 (above). Framework for CFS research to enhance, inform, and complement current reclamation practices for re-establishing healthy, productive forest ecosystems Institutional arrangements and policy instruments focused on oil sands related issues (e.g., tailings ponds) identified by stakeholders to which CFS science can contribute to determine suitable reclamation practices to mitigate the impacts

2.4.7 Distribution of components by project area

Considering these activities and outputs, the six FESA Program Sub-Activity project areas supported a total of 502 components between 2007-08 and 2011-12 (see Table 8).^{Footnote 36} A component can be any type of activity that contributes to the overall intended outcome. Examples of common components include scientific research studies, working groups, development of tools or new networks, mapping activities, knowledge translation, development of online services and more.

Table 8 indicates an overall decrease in the number of components across all projects areas since 2008-09 (except for Land Reclamation, since it was launched in 2011-12). This reduction occurred as a result of ongoing efforts from program management to refocus and streamline projects by ensuring that similar work conducted across CFS centres was grouped into one component, rather than having individual components for each centre. The intent of this exercise was to increase coordination, efficiency and collaboration across CFS centres.

Forest Biodiversity has the highest number of components, with almost a third of the total number. On the other hand, IMFN and Land Reclamation have fewer components when compared to the other project areas (47 and 6 components, respectively).

Table 8 Number of components by project area, 2007-08 to 2011-12

Fiscal Year	Biodiversity	Carbon	Productivity & Dynamics	IMFN	NFIA	Land Reclamation
2007-08	-	-	-	-	-	-
2008-09	41	36	30	-	38	-
2009-10	30	29	34	10	22	-
2010-11	39	30	21	8	18	-
2011-12	18	16	8	7	7	6
Total	128	111	93	25†	85	6

Note: † The three AMFI components (one per year) are included in the total; however these components were not examined as part of this evaluation.
Source: NRCan. ProMIS

2.5 Funding and Resources

Table 9 presents the total resources for the FESA Program Sub-Activity per project area during the evaluation period (2007-08 to 2011-12), including actual internal expenditures (A-base and C-base) and external support (actual financial and planned in-kind). As part of C-base funding, the Sub-Activity received funding for forest policy, monitoring and capacity-building activities as part of horizontal initiatives funded by the Government of Canada. These include, the Mackenzie Gas Project, Clean Air Agenda, and International Polar Year.

Table 9 Summary of internal and external expenditures, by FESA project area and overall, 2007-08 to 2011-12 ($ millions)

Project area	Internal resources (actual)			External resources		TOTAL (actual + planned in-kind)
	A-base	C-base	Internal TOTAL*	Financial (actual)	In-kind (planned)**
Biodiversity	16.6	1.0	17.7	0.5	2.9	21.1
Carbon	12.3	6.8	19.1	0.3	0.4	19.8
IMFN***	1.9	4.4	6.4	0.9	0.5	7.7
NFIA	21.8	1.2	23.1	3.2	6.3	32.6
Productivity & Dynamics	16.8	0.4	17.2	0.7	1.3	19.3
Land Reclamation	0.8	0.1	0.9	0.1	0.1	1.1
TOTAL	70.3	14.1	84.4	5.7	11.4	101.5

Notes: Totals may be ±0.1 due to rounding.
* Internal (NRCan) funding includes A-base and C-base, and covers salary, operating and maintenance, G&Cs and transfer payments.
** Data on in-kind contributions are not available for 2007-08.
*** Figures exclude G&C funding for the African Model Forest Initiative because the AMFI is part of a separate evaluation. The IMFN became part of the FESA Program Sub-Activity in 2009-10 and these figures do not include prior funding.
† The land reclamation project area started in 2011-12.
Source: Compiled by NRCan. Internal and external resources, Actual Expenditures Reported by FESA; In-kind contributions, ProMIS.

Table 10 presents the expenditures by type – Operating and Maintenance (O&M), Salary, and Grants and Contributions (G&C) – for the FESA Program Sub-Activity by project area during the evaluation period (2007-08 to 2011-12). Note that this table does not include the in-kind resources presented in Table 10, such that the total is reduced by $11.4 million, from $101.5 to $90.1 million.

Table 10 FESA Financial Expenditures by type and by project area, 2007-08 to 2011-12 ($ millions)

Expenditure type	Biodiversity	Carbon	IMFN	NFIA	Productivity & Dynamics	Land Reclamation	TOTAL
O&M	2.5	3.2	2.0	7.9	2.4	0.2	18.1
Salary	15.7	16.2	2.1	18.4	15.6	0.8	68.9
G&C	0	0.1	3.1	0	0	0	3.1
Total	18.2	19.4	7.2	26.3	18.0	1.0	90.1

Note: Both O&M and Salary expenditures include A-base, C-base and External; exclude in-kind
Source: Forest Ecosystems IO

2.6 FESA Logic Model

The Program Sub-Activity-level logic model that outlines the program theory is presented in Figure 1. This logic model was developed by the FESA Program Sub-Activity (also referred to internally as the Forest Ecosystems Intended Outcomes), and adapted for the evaluation.

The first row of the logic model identifies the main types of financial resources (inputs) that support the activities, outputs and outcomes of the FESA Program Sub-Activity outlined in the following rows.

3.0 Evaluation Objectives, Scope, and Methodology

3.1 Objectives and Scope

This evaluation was mandated by NRCan’s Evaluation committee in accordance with the Financial Administration Act and the 2009 Treasury Board Policy on Evaluation^{Footnote 37}. The primary objective of the evaluation was to assess issues relating to the relevance and performance of the FESA Program Sub-Activity and provide recommendations as necessary for the period from 2007-08 to 2011-12.

All six project areas were covered by the evaluation, with the exception of the African Model Forest Initiative (within the IMFN project area). The evaluation followed a theory-based approach,^{Footnote 38} in which emphasis is placed on the process by which the program arrives at its expected results (see further details in the next section).

This evaluation addresses the five core evaluation issues defined by the Treasury Board Secretariat in the Directive on the Evaluation Function, effective April 2009:

Relevance of the FESA Program Sub-Activity:

• Continued need for Program (R1)
• Alignment with government priorities (R2)
• Alignment with federal roles and responsibilities (R3)

Performance (effectiveness, efficiency and economy) of the FESA Program Sub-Activity:

• Achievement of expected outcomes (P1 & P2)
• Demonstration of efficiency and economy (P3 & P4)

These five evaluation issues are addressed through a series of specific evaluation questions, which are answered in Section 4.0.

3.2 Methods

The evaluation questions and the selection of methods were informed using a theory-based approach and were developed in consultation with the Evaluation Advisory Committee. A theory-based approach was recommended by the evaluation assessment to structure and undertake the analysis in this evaluation as an alternative to an experimental design evaluation. While an experimental design typically measures both the baseline and the final results associated with an intervention by incorporating comparison groups, “theory-based approaches to evaluation use an explicit theory of change to draw conclusions about whether and how an intervention contributed to observed results”.^{Footnote 39}

Accordingly, this evaluation placed more emphasis on the process by which the program arrives at its expected results using a theory of change to help track the linkages and contribution of outputs to outcomes and to tell the story of performance. A theory-based approach to evaluation is particularly useful for the assessment of a Sub-Activity being evaluated for the first time as it helps understand why and how the observed results occurred in various contexts. Using this conceptual analytical model, the evaluation was designed to draw on various sources of data to ensure that the combined lines of evidence resulted in an in-depth and comprehensive analysis. A data collection matrix was used to cross-link evaluation questions with associated performance indicators, data collection methods and data sources, allowing data to be triangulated.

Five methods were used to collect and analyze evidence. Table 11 presents details on each data collection method.

Table 11 Overview of evaluation data collection methods

Data Collection Method	Details
1. Document, file and data review	A review of approximately 200 documents, secondary literature, files and Sub-Activity data including: Departmental documents (strategic plans, annual reports, audits, etc.) Program documents (financial and administrative databases, performance monitoring, communication outputs, etc.)
2. Database review	A review of CFS ProMIS planning database conducted by NRCan-SED
3. Stakeholder interviews and focus groups	Total number of interviews.................................................................... 97 Internal (FESA management and staff) ............................................... 29 External (other NRCan staff, OGDs, P/Ts, academia, industry and other stakeholders)..................................................................................... 68 Other NRCan staff......................................................................... 8 Other Government Departments (OGDs) ..................................... 11 Provinces/Territories (P/Ts) ........................................................ 20 Academia.................................................................................... 10 Industry......................................................................................... 6 Environmental Non-Governmental Organizations (ENGOs) ............. 3 Other international representatives................................................. 5 IMFN international representatives................................................. 5 Total number of focus groups................................................................. 5 Number of participants (P/T representatives, NRCan policy staff)......... 31 National Capital Region ................................................................. 8 British Columbia ........................................................................... 7 Ontario ......................................................................................... 6 Quebec ........................................................................................ 6 Alberta ......................................................................................... 4
4. Survey	Online survey of project area component leads 18 questions total Total survey population (N) ................................................................ 93 Number of responses (n) .................................................................... 56 Response rate (†).......................................................................... 60.2% Margin of error (‡) .......................................................................... 8.3%
5. Bibliometrics and webmetrics	Bibliometric analysis Peer-reviewed papers ................................................................ 123,416 Grey literature/CFS documents ...................................................... 4,521 Webmetric analysis of project area publication and non-publication outputs Using analysis of citing documents and websites

Note: † Valid response rate = Number of completed surveys, divided by the valid sample, which excludes unreachable potential respondents; ‡ Calculated for a response distribution of 50% (i.e., 50% yes/50% no); 95% confidence level (19 times out of 20)

3.3 Evaluation Challenges, Limitations and Mitigation Strategies

The challenges and data limitations encountered during this evaluation, and the mitigation strategies adopted to counter them, are discussed below. Generally, challenges were anticipated early in the process and associated mitigation strategies were proactively built into the evaluation design. A key mitigation strategy used was the comparison of information obtained from different data collection methods to validate findings.

3.3.1 Challenges

Broad scope of the sub-activity: The broad scope of FESA components, spread across six different project areas meant that a large range of different stakeholder groups had to be consulted, both within and outside the federal government. While this presented some difficulties in discerning the commonalties within groups, this issue was mitigated to a large extent through the use of qualitative data analysis software.

Multiple/diverse roles of internal stakeholders: Some interviewees and focus group participants representing NRCan (but outside the FESA) were sometimes both users and producers of FESA information, which posed challenges to mapping of outputs/outcomes and feedback within FESA overall and for specific project areas. To mitigate potential for bias, interviewees/focus group participants were asked to differentiate when speaking from a user vs. a producer perspective.

Awareness of sub-activity: Some external stakeholders had difficulties differentiating FESA activities from other CFS activities even though mitigation strategies were implemented such as using specific prompts.

Evaluation design: The granularity of the information required in the data collection matrix did not allow for distinct coverage of all questions and indicators. As a result, interview questions were combined in these cases and some questions were not covered in all interviews.

In addition, the distinction between some intermediate and ultimate program outcomes was found to be conceptual rather than operational. As a result, some answers to some interview questions were combined. This was the case particularly for performance questions.

Fieldwork challenges: During the fieldwork phase it was challenging to schedule nearly 100 interviews (and five focus groups across Canada) and analyze data collected through this method within a short timeline.

3.3.2 Data limitations

Data gaps: An apparent data gap arose on the issue of forest ecosystem risk. Two reasons may explain this gap: 1) indicators for these two questions were not as comprehensively covered in the interview/focus group guides and 2) interviewees tended to focus on the contribution of FESA to their everyday activities rather than on identification/ understanding/management of risks.

In addition to this general data gap, there are also data limitations associated with each data collection method:

Survey: The survey was only administered to internal stakeholders, in particular representatives of individual project areas that produce FESA outputs. To mitigate, the evaluation extensively consulted external stakeholders via the interview and focus group methods. Cautious interpretation of survey results should also be exercised given the survey’s relatively high margin of error (8.3%).^{Footnote 40}

Document review: The availability, quality and comprehensiveness of documents across project areas were highly uneven. Particularly, the evaluation team was faced with a lack of performance measurement data such as quarterly or annual progress reports or limited evidence of the implementation of a performance measurement strategy. To the extent possible, performance measurement information was collected through interviews and the survey.

Bibliometric analysis: The bibliometric framework used for this evaluation does not aim to be a complete inventory of FESA’s output. The keyword-based searches may not capture all relevant documents. In addition, the method also has some inherent limitations such as that the bibliometric databases tend to have a slight bias for countries that publish in English-language journals. Although French was included in the analysis, it is possible that other countries may be underestimated. Lastly, different citation patterns may be observed across disciplines, which may complicate their comparison.

Database review: The CFS ProMIS is a planning database and as such does not include any reporting on outcomes and financials. As a result, actual expenditures were reported separately by FESA. Relevant information from the two datasets needed to be manually combined into a Microsoft Excel spreadsheet in order to enable the analysis of data.

4.0 Evaluation Findings

4.1 Relevance

4.1.1 Continued Need for the Program (Issue 1)

Question 1: Is there an ongoing need for the Program Sub-Activity?

FESA remains highly relevant to address several environmental and economic needs, such as compiling and reporting national information on the state of Canada’s forests, providing expertise to address the impact of climate change and facilitating market access for Canadian forest products. FESA’s specialized and long-standing scientific expertise inevitably fills a gap, addressing topics not covered by academia, provincial governments or other organizations.

Specific ongoing needs were also identified within individual FESA project areas. Notably, there is a need for CFS to clearly identify its niche within the Land Reclamation area considering the complexity of the topic and the many players already involved.

Identified needs: All lines of evidence confirmed that FESA remains highly relevant to address several ongoing environmental and economic needs. On the economic side, the forestry sector in Canada employs some 600,000 people (direct and indirect jobs), and supports approximately 200 Aboriginal and forest-based communities.^{Footnote 41} At the same time, the international market context for Canadian forest products has changed dramatically during the last decade. Market access is now strongly influenced and facilitated by Canada’s reputation as a global leader in sustainable forest management (SFM).^{Footnote 42 ,Footnote 43} To maintain this position, Canada must draw on balanced and credible scientific expertise to support policy direction and to enable participation in international forums on related issues.^{Footnote 44} While the FESA sub-activity does not involve direct market access activities, there is an ongoing need for the sub-activity as it directly supports and links to the overall NRCan objectives relating to the forest resource sector.

On the environmental side, a need to better understand climate change impacts and climate change mitigation potential, as well as to preserve national biodiversity, are supported through FESA activities, as outlined throughout the Performance section of this report. Furthermore, Canada’s forests are a large store of carbon and important in the global carbon cycle^{Footnote 45} – thus understanding how management activities and natural disturbances affect the carbon is particularly relevant in the context of addressing global climate change.^{Footnote 46}

Unique expertise: Having identified these ongoing needs, FESA provides the unique and long-standing expertise in forest ecosystem science that is particularly relevant to address climate change mitigation and adaptation issues and to make decisions that support forestry sector competitiveness. Internal and external interviewees also widely referred to the need for scientific information that would help maintain or generate social acceptance of forest harvesting and management practices within and outside Canada to facilitate market access.

An analysis of the specialization pattern of Canadian sectors across FESA project areas further suggests that FESA provides unique scientific expertise in the Canadian forestry context. Indeed, compared to other Canadian sectors, the 2003-2006 and 2008-2011 analyses (see Figure 2) indicate that CFS and the federal government allocate a much greater share of their forestry output, relative to the world, in the Forest Assessment^{Footnote 47} and Carbon^{Footnote 48} project areas. These findings show that the federal government and CFS (especially in recent years) have been filling a gap in areas that do not constitute the core activities, in relative terms, of academia and provincial governments.

Biodiversity is the only area in which none of the sectors is specialized. In 2003-2006, nearly all sectors were devoting an equal share of their forestry output to Biodiversity as the world was (i.e., SI = 1), the only exception being CFS (and the federal government) with a smaller share than the world (SI < 1). In 2008-2011, the specialization of most sectors decreased very slightly. Only CFS (or the federal government) increased its specialization in Biodiversity approaching world level. This increase in specialization is attributable to a faster increase in the share of its forestry output (or that of the federal government) allocated to Biodiversity than observed at world level; it reached 23% in 2008-2011 compared to about 17% in 2003-2006. This is the strongest increase in the concentration of CFS (or the federal government’s) forestry output in a given area. Nevertheless, Biodiversity remains the only area in which Canada does not have a specialization above world level in 2008-2011; it even decreased slightly relative to 2003-2006.

Project areas: Ongoing needs specific to individual project areas were also identified in reviewed documentation and during interviews/focus groups. For instance, within the Forest Assessment project area there was wide consensus on the need to maintain and enhance the National Forest Inventory (NFI), a component which provides important scientific information to the Government of Canada. FESA’s role was deemed critical to integrate and harmonize data produced by provincial jurisdictions to provide an overarching, national picture of the state of Canada’s forests. Moreover, the NFI generates baseline information on forests that informs research activities carried out within all other FESA project areas. It also informs a number of other CFS activities to enhance Canada’s forest management practices.^{Footnote 49} The ability to provide credible information on sustainable forest management also influences Canada’s position in the competitive forest products market and thus relates to broader NRCan objectives as discussed above.^{Footnote 50}

In the Carbon project area, FESA also plays a significant role, as it provides research and policy advice, the demand for which is expected to increase within the federal government. As an example, CFS recently contributed to Environment Canada’s report on GHG reduction potential, estimating reduction potential for different land use scenarios.^{Footnote 51} It is expected that Land Use, Land-Use Change and Forestry will have an increasing impact on global carbon accounting policies and practices,^{Footnote 52} an area for which CFS is responsible. In addition, following the termination of the Canadian Carbon Program in 2011, NRCan has indicated that CFS should play the lead role in coordinating further Canadian carbon science. ^{Footnote 53}

Similarly, the Land Reclamation project area is meant to fill significant research and policy gaps in this relatively new topic.^{Footnote 54} This project area focuses on the Alberta oil sands, a complex issue that requires CFS to adopt a highly collaborative approach within NRCan, the Government of Alberta and many other forest sector players that are already active in this field.^{Footnote 55} To avoid duplication of effort, CFS must clearly identify which research questions it can best address and allocate funding and human resources accordingly.

The project area entitled Ecosystem Integrity unified two existing project areas (Biodiversity and Productivity & Dynamics) in 2011-12 to better address policy needs. The new project area uses biodiversity expertise to focus on how forest management affects forest properties, through the development of ecosystem integrity indicators.^{Footnote 56} However, while this restructure helped clarify and focus the scope of research conducted within these two project areas, interviewees noted that some biodiversity scientists were excluded as their work was no longer aligned with the objectives of the new project area. On the other hand, the new project area was generally considered successful in ensuring strong science-policy integration, given the reasonably limited resources available.

These findings may indicate an ongoing need for sustained efforts to ensure that scientists (primarily in the regions) and the policy makers meet regularly and work collaboratively towards the fulfillment of policy priorities related to forest ecosystem research. This could be achieved by ensuring, at the beginning of each project, that scientists clearly articulate the expected impact of their work on policy.

Finally, evidence from the evaluation confirms that participation in the IMFN initiative is increasing worldwide, which demonstrates an ongoing need for the network. Membership has grown to 60 Model Forests in more than 30 countries. ^{Footnote 57} Canadian Model Forests and IMFN have brought diverse stakeholder groups (e.g., industry, government, First Nations, community-based associations, ENGOs, educational and research institutions and private landowners) together to generate new ideas and on-the ground solutions to sustainable forest management issues.^{Footnote 58} Interviews with internal and external stakeholders confirmed that there is an ongoing important need for IMFN and that this project area currently adequately meets needs, given the current level of FESA resources. IMFN stakeholders suggested that given the wide regional experience of the various Model Forests, IMFN should consider exploring themes that could apply “globally” such as governance, integrated landscape management, climate change, building local capacity, etc. NRCan stakeholders note that IMFN itself is a governance model in the service of “integrated landscape management.” The network meets every three to four years to discuss global themes. In 2011, the network agreed upon the following global issues: Climate Change, Environmental Goods and Services, Community Sustainability, and Knowledge Management.

Moreover despite the fact that stakeholders believe there is a need for IMFN, this project area was rated^{Footnote 59} a lesser priority than the other project areas by science and policy experts surveyed as part of a CFS 2011 Environmental Scan.^{Footnote 60} However, the report highlights that, despite priority ranking, “there was strong support for the activities that could be expected from a model forest: Work and training in the application of ecosystem-based forest management would significantly advance the understanding and application of this approach (58% of respondents); and Informal networks of forest areas piloting and demonstrating the application of ecosystem-based forest management would significantly advance the understanding and application of this approach (55% of respondents).”

4.1.2 Alignment with Government Priorities (Issue 2)

Question 2: Is the Program Sub-Activity consistent with government priorities and NRCan strategic outcomes?

FESA’s expected outcomes have been clearly aligned with federal priorities and NRCan’s strategic objectives throughout the evaluation period. Namely, these include the development of a competitive forest industry and responsible natural resource management. In terms of the six project areas, there is less direct alignment between IMFN and FESA. However, a scan of the NRCan PAA indicates that the IMFN contributes to – or could potentially be relevant for – other Sub-Activity Programs under the responsibility of CFS.

Federal priorities: The objectives of the FESA Sub-Activity overall, and the objectives of most individual project areas are highly aligned with the priorities of the federal government and the strategic objectives of NRCan. In fact, documentary evidence such as Budgets,^{Footnote 61} Speeches from the Throne^{Footnote 62} and the Economic and Action Plan^{Footnote 63} suggests that GoC priorities have included development of a competitive forest industry for at least the last decade.

Departmental outcomes: FESA is also consistent with NRCan’s mandate to create a sustainable resource advantage within departmental Strategic Outcome # 2 “Natural Resource Sectors and Consumers are Environmentally Responsible”.^{Footnote 64} Under NRCan’s 2012-2013 Program Activity Architecture (PAA), FESA is aligned with PA 2.3 “Responsible Natural Resource Management, which seeks to ensure that “public and private sectors establish practices to mitigate the environmental impacts to natural resources”.

Among the six FESA project areas, there is less direct alignment between IMFN and FESA Sub-Activity. The Sub-Activity’s stakeholders consulted perceived that the domestic mandate of FESA does not clearly parallel the international goals of the IMFN. However, a scan of the NRCan PAA indicates that the IMFN contributes to – or could potentially be relevant for – other Sub-Activity Programs. In fact, as the IMFN is envisioned to lead in developing the international network “as a platform to import knowledge and experiences gained elsewhere to aid in developing various solutions to forest management challenges in Canada^{Footnote 65} [65]”, the IMFN objective is relevant to more than one single Sub-Activity Program, including FESA and other Sub-Activity Programs under the responsibility of CFS, including (but not limited to):

• Positive perception of Canadian forest practices and products among targeted stakeholders in key international markets (Sub-Activity 1.1.2 : Forest Products Market Access and Development,^{Footnote 66} and;
• Forest-based and Aboriginal communities have the knowledge needed to take advantage of emerging economic opportunities: Sub-Activity 1.3.2: Forest-based Community Partnerships.^{Footnote 67}

In addition, a recent independent study^{Footnote 68} commissioned by NRCan to inform the decision-making process about the IMFN secretariat confirmed existing and potential links between the IMFN and the Government of Canada agenda. In its recommendations, the study highlights the value proposition of the IMFN to the GoC:

• “The most attractive value proposition of the IMFN to the GoC is based on the contribution of the network to Canada’s declared objective of sustainable prosperity from natural resource development. Model Forests can show how an important natural resource can be effectively managed in the context of multiple objectives –environmental, social and economic – at the same time contributing to the fulfillment of Canada’s international obligations with respect to climate change and sustainable natural resource development.”

4.1.3 Alignment with Federal Roles and Responsibilities (Issue 3)

Question 3: Is there a legitimate, appropriate and necessary role for the federal government in these project areas?

Although forests generally fall under provincial/territorial jurisdiction, the role of the federal government is legitimate to conduct forest ecosystem research and develop a national and long-term perspective supported by science-based information. The Government of Canada is also uniquely positioned to represent Canada’s interests globally and to address international obligations.

Furthermore, CFS’ output in forest ecosystem research, relative to provinces/territories and academia, suggests that it complements the roles of others and several examples of synergistic work were identified.

However, there is an opportunity to re-examine the role for CFS in the IMFN project area to better align with the FESA mandate, and to clarify its role in Biodiversity research. Considering the fact that the bibliometric analysis indicated that none of the other sectors (i.e., academic and provincial government sectors) are specialized in forest biodiversity research, the recent increase in specialization of CFS in Biodiversity research (the strongest increase in the concentration of CFS’ forestry output in a given project area), and the ongoing need for Biodiversity science to help support international commitments, it appears to be appropriate that the federal government maintain its role and expertise in this area. CFS plays an important role in supporting biodiversity relevant to forestry.

In the case of the IMFN, given that it is unlikely to find an external host (without substantial financing accompanying such a transfer) there is an opportunity for CFS to deliberately engage the department regarding opportunities for scientific research, knowledge mobilization on sustainable management of forest-based landscapes and for other relevant actionable issues.

To clearly manage provincial expectations and acknowledge long-standing relationships, there is a need for FESA to regularly communicate any changes in its expertise and services because of increasing focus on federal priorities.

Legitimate role: While more than 70% of Canada’s forests are administered by provinces and territories, the federal government remains responsible for areas under national jurisdiction, including economy, trade, international relations, science, technology and the environment.^{Footnote 69} FESA’s mandate relating to the production of scientific knowledge for sustainable forest management and the coordination of information at the national level is aligned with the above federal responsibilities. Therefore, it is legitimate for the federal government to be delivering FESA activities.

Moreover, the FESA sub-activity is consistent with the duties and responsibilities executed by the Minister of Natural Resources under the Natural Resources Act (1994) and the Forestry Act (1985). Indeed, as per the former, the Minister shall, “seek to enhance the responsible development and use of Canada’s natural resources and the competitiveness of Canada’s natural resource products”.^{Footnote 70} The Forest Act outlines NRCan’s specific responsibilities related to the protection and wise management of Canada’s forests and forest resources, which involve the conduct of research activities and the provision of advice to stakeholders.^{Footnote 71}

National perspective and international representation: The legitimacy and necessity of the role of the federal government in FESA project areas was confirmed by views expressed by interviewees and focus groups. There was a clear consensus across stakeholder groups that the federal government’s involvement is critical to convene a national and long-term perspective on forest ecosystems, through the integration of provincial and external data. The National Forest Inventory (NFI) was a frequently-cited example of such an activity. The federal role is all the more important considering that provinces and territories alone do not have the capacity (e.g. resources, time) to deliver a program with the broad FESA scope. Similarly, interviewees believed that the academic sector does not have the mandate to coordinate national scientific work, nor is it driven by policy priorities.

Besides meeting national obligations, part of the federal government’s role is to represent Canadian interests abroad and fulfill its international commitments. Indeed, such responsibility cannot be endorsed by the provinces and territories themselves, since they are not recognized as legitimate interlocutors (that is to say, representatives of Canada) during the negotiations, discussions and with respect to reporting requirements. In fact, many interviewees noted that without federal involvement, Canada would lose its status as a leader in areas such as Carbon and IMFN. Through CFS and FESA, Canada remains active at the international level, particularly in the context of the UN Convention on Biological Diversity, the UN Framework Convention on Climate Change (UNFCCC) and the preparation of the annual Environment Canada GHG Report which feeds into international reporting under the UNFCCC.

Complementary and synergistic role: Considering the work of other players in the field, the evaluation also found that FESA plays a complementary role. For example, bibliometric analyses show that CFS makes a significant contribution to forestry scientific production. Indeed, a quarter of Canadian peer-reviewed publication output in forestry research involved at least one federal researcher and 16% of Canadian peer-reviewed papers in forestry were authored by CFS scientists (Table 12). CFS work involves a strong collaborative effort as well, since more than two thirds of these papers (65%) were co-authored with scientists from other Canadian institutions. Moreover, in a large network of organizations producing scientific publications in forestry, CFS clearly plays a central role as its scientists constitute additional resources that enrich and complement the work of other research organizations.

Table 12 Peer-reviewed scientific output in Forestry, 2003–2011

Entity	Papers		Share of Canada	Share of World
	No.	GI	%	%
World	123,416	1.42	n.a.	100.0
Canada	10,671	1.20	100.0	8.6
Academic	8,637	1.27	80.9	7.0
Government	3,541	1.12	33.2	2.9
Federal Government	2,607	1.11	24.4	2.1
Provincial Government	1,194	1.14	11.2	1.0
Industry	601	0.84	5.6	0.5
Not-for-Profit Research Organization	359	0.94	3.4	0.3
Others	250	1.35	2.3	0.2
Natural Resources Canada	1,847	1.09	17.3	1.5
Canadian Forest Service (CFS)	1,711	1.12	16.0	1.4

Note: The GI measures the increase/decrease in output between the first (2003–2006) and second half (2008–2011) of the study period.
Source: Computed by Science-Metrix using the WoS

In addition to the quantitative data, there was a wide consensus across interview groups that there exists little or no overlap between the work of CFS and other research organizations, government departments and jurisdictions. Clear synergies exist in FESA’s contribution to Environment Canada’s GHG inventory, among provincial governments (e.g. for the Forest Assessment and Reclamation project areas), and with academic researchers (e.g. regarding work on the mountain pine beetle or fire modelling).

Opportunities for improvement: Despite a clear role for the federal government overall, there may be an opportunity to re-examine CFS role for two project areas, namely Biodiversity and IMFN.

Biodiversity: Responsibility for biodiversity in Canada is shared among federal departments and agencies, including Environment Canada, Fisheries and Oceans Canada, Agriculture and Agri-Food Canada, Natural Resources Canada, and the Parks Canada Agency; and between federal, provincial, and territorial governments.^{Footnote 72} The Government of Canada, with support from the provincial and territorial governments, signed and ratified the United Nations Convention on Biological Diversity in 1992. Subsequently (1995) a Canadian Biodiversity Strategy was created which included key objectives of establishing protected areas and improved forest management.^{Footnote 73} Environment Canada is the National Focal Point for Canada for the Convention on Biological Diversity. The overarching governance mechanism for Biodiversity is the federal/provincial/territorial Canadian Council of Resource Ministers.^{Footnote 74} There is also a federal/provincial/territorial Biodiversity Working Group that focuses on national biodiversity issues. This working group also supports the Canadian Council of Resource Ministers.

The bibliometric analysis (see Figure 2) indicates that, unlike what was observed in other project areas, none of the main Canadian sectors producing forestry papers were specialized in Biodiversity research in the time periods 2003-2006 and 2008-2011. In fact, Biodiversity remains the only area in which Canada did not have a specialization index (SI) above world level in 2008-2011; the SI even decreased slightly for Canada as a whole and most sectors.

Nevertheless, this analysis also shows that the level of specialization of both CFS and the federal government in Biodiversity increased during the period 2008-2011 in comparison to 2003-2006 while slight decreases were observed in other sectors. This change is attributable to an increase in the concentration of CFS’s forestry output in Biodiversity (and concomitantly at the federal government level) that exceeded the one observed at world level. In fact, this is the strongest increase in the concentration of CFS’ (or federal government) forestry output in a given area.

There is a risk that this trend will not be maintained in the longer term as some of the Biodiversity research expertise developed over the last five years is not likely to be pursued into the new Ecosystem Integrity project area. In turn, the declining trend in Canada’s specialization in Biodiversity research could be accentuated, eventually leading to a share of Canada’s forestry output devoted to this area smaller than at the world level. The outputs generated in Biodiversity are highly relevant towards meeting Canada’s obligations under the UN Convention on Biological Diversity, the Species at Risk Act and the Canadian Biodiversity Strategy. For CFS, forest biodiversity research supports the delivery of knowledge and advice to the forest sector that would strengthen Canada’s ability to demonstrate the science based foundation of Canada’s forest policy.^{Footnote 75}

Considering the fact that none of the other sectors are specialized in this area, the recent increase in specialization of CFS in Biodiversity research, and the ongoing need for Biodiversity science to help support international commitments, it appears to be appropriate that the federal government maintain its role and expertise in this area. While biodiversity is a shared responsibility among federal, and provincial/territories governments, CFS plays an important role in supporting biodiversity research relevant to forestry. Therefore, there is an opportunity to further examine whether biodiversity expertise has been adequately integrated into the new project area – Ecosystems Integrity.

While the above findings indicate a need for CFS to remain involved in Biodiversity research, some interviewees noted that Environment Canada could carry out some programming related to this project area, (e.g. species-at-risk issues). However, according to interviewees EC’s current mandate and resources may limit its ability to actually absorb these activities. For instance, CFS has traditionally been responsible for conducting caribou research since these animals are an indicator of forest ecosystem health. On the other hand, there is a risk that if EC were to conduct caribou research for species-at-risk programming the forest ecosystem perspective would be diminished or lost; a perspective for which there is a continued need.

IMFN: IMFN interviewees suggested that other organizations, such as the FAO or other federal departments, may be better placed to deliver the IMFN, given the less direct alignment of its objectives with the FESA and CFS mandate. Recognizing the challenges that would come with moving the IMFN, most stakeholders indicated that Canada could maintain a role in providing financial and administrative support, at least through the transition period. However, some IMFN interviewees argued that, given that it is unlikely to find an external host (without substantial financing accompanying such a transfer)^{Footnote 76} , CFS should undertake deliberate activities to formally position the initiative within NRCan and FESA.

A recent independent study^{Footnote 77} commissioned by NRCan^{Footnote 78} has also clarified that the transfer of the IMFN secretariat outside of CFS is not a viable option. This study recommended that “IMFN should continue to be housed at NRCan – the costs are modest and the benefits are real.”

There may be an opportunity to clarify federal and provincial roles under the FESA Sub-Activity. Focus group evidence indicates that the provinces and territories expect that CFS will continue to address their needs and fulfill capacity gaps. They noted that there is sometimes a lack of communication from the federal government on what information it (the federal government) needs collected. As well, there is a lack of coordination among the provinces which means that consistent and comparable data is not always provided for national reporting. At the same time, CFS is increasingly focussing its activities on federal needs and priorities such as science-policy integration. From an objective standpoint, it would appear that there is a clear opportunity here to improve and align federal and provincial communication processes.

4.2 Performance

4.2.1 Achievement of Expected Outcomes (Issue 4)

Question 4: To what extent have Canadians and their institutions had access to authoritative scientific information on forest ecosystems at national scale? (P1.1)

All six projects areas under the FESA Program Sub-Activity generated a wide array of outputs, which have been accessed and used by relevant stakeholder groups. The Sub-Activity’s stakeholders were generally satisfied with the accessibility of FESA outputs; however, some opportunities for improvement were identified with respect to the communication and usability of these outputs and the need for more direct interactions with stakeholders.

Target audience and generation of outputs: All lines of evidence confirmed vast production, access and use of FESA outputs by a wide range of target audiences, across five of the FESA project areas. As Land Reclamation has been launched recently it has yet to generate as many outputs as the other project areas.^{Footnote 79} Based on the survey of component leads, outputs such as knowledge, information, and expertise were generated to a large or great extent by more than, or close to, three-quarters of FESA components (Figure 3). The primary target audiences for these outputs were other NRCan CFS programs, provinces and territories, industry, academia and other federal government departments (Figure 4).

In more detail, interviewees and focus groups cited various examples of the kinds of outputs frequently used by target audiences. These include the:

• website of the Canadian Council of Forest Ministers (CCFM).^{Footnote 80} This provides a broad spectrum of accessible information including video vignettes and fact sheets on various topics pertaining to SFM;
• National Forest Inventory’s (NFI) website.^{Footnote 81} Operated by the CCFM, this allows the forest community and broader public to access reports, data, maps and statistics on the state of Canada’s forests. It is noteworthy that the NFI has been improved over the last decade and now includes a comprehensive set of 46 of economic, environmental and social values of Canadian forests;
• National Forestry Database,^{Footnote 82} a website that provides information on forest management activity in Canada;
• Canadian Model Forest Network’s (CMFN)^{Footnote 83} contribution to boreal forest and wildlife research, and best practice toolkits and guidelines from the IMFN^{Footnote 84} in addition to events aiming to share information, progress and achievements of regional networks.^{Footnote 85}
• Carbon Budget Model (CBM),^{Footnote 86} used both nationally by provinces/territories representatives and internationally by other countries; and the
• annual State of Canada’s Forest reports^{Footnote 87} and the Canadian National Vegetation Classification website^{Footnote 88} used as key reference documents by a range of stakeholders.

FESA/CFS also carries out S&T activities with industry and academia that generate traditional research products. As illustrated in Table 13, more than half of the total CFS document output in forestry, as well as in each of the selected FESA project areas, encompasses peer-reviewed papers published in scholarly journals.

Comparing the output of each of the project areas (Figure 5), it is noteworthy that Productivity & Dynamics is responsible for about a quarter of all FESA peer-reviewed and grey literature outputs. The distribution of outputs per project area reflects the wider scope of the respective areas. For instance, Productivity & Dynamics covers the entire spectrum of forest ecosystems-related issues, as opposed to Carbon which focuses on one particular aspect.

Table 13 Canadian Forest Service Publication Output by FESA project area and Document Type, 2003-2012

Document type	CFS Output		FESA Project Area (%)
	No.	% of Total	Biodiversity	Carbon	Productivity & Dynamics	Forest Assessment
All types	4,521	100.0	10.0	6.0	27.6	11.0
Book chapters	176	3.9	2.7	4.0	4.2	4.0
Contributions to proceedings	282	6.2	5.1	6.6	5.3	10.3
Journal Article	2,431	53.8	66.3	66.9	53.2	56.1
Miscellaneous items	92	2.0	1.6	1.8	1.2	2.6
Monographs	440	9.7	5.3	5.9	9.0	6.2
Posters	27	0.6	0.4	0.7	0.5	1.0
Proceedings	15	0.3	0.0	0.0	0.2	0.6
Series Item	1,058	23.4	18.6	14.0	26.3	19.1

Source: Compiled by Science-Metrix from the list of FESA outputs

There is also variability among the contribution of each forestry centre to the FESA outputs. The Pacific Forestry Centre accounts for one quarter of all the outputs and its contribution is highest in the Forest Assessment project area. On the other hand, the Northern Forestry Centre specializes in Carbon research as its contribution under this project area is 2.5 greater than CFS average and three to four times greater than most other centres. The ratios of outputs by each CFS centre are further depicted in Table 14.

Table 14 Canadian Forest Service publication^{Footnote 90} output by FESA project area and CFS research centre, 2003–2012

CFS research centre	CFS Output		FESA Project Area (%)
	No.	% of Total	Biodiversity	Carbon	Productivity & Dynamics	Forest Assessment
CFS	4,521	100.0	10.0	6.0	27.6	11.0
Atlantic Forestry Centre	443	9.8	12.9	4.5	24.2	5.4
Great Lakes Forestry Centre	706	15.6	6.5	5.5	29.2	5.9
Laurentian Forestry Centre	990	21.9	13.2	4.4	22.7	4.4
National Capital Region	537	11.9	8.8	2.4	40.8	7.6
Northern Forestry Centre	674	14.9	12.6	15.7	34.7	10.7
Pacific Forestry Centre	1,177	26.0	7.3	4.4	22.0	23.4

Source: Compiled by Science-Metrix from the list of FESA outputs

User access and satisfaction: The outputs, expertise and services provided under this Sub-Activity are primarily accessed via personal interactions with CFS staff, collaboration on joint projects, remotely in case of tools available online on the NRCan website (note that more than half of surveyed component leads indicated their outputs were available online) and through the NRCan File Transfer Protocol. Outputs are also accessed via committees, meetings, conference calls and workshops, as well as peer-reviewed papers (downloaded or shared by FESA scientists).

In addition, a webmetric analysis showed that documents published by the selected FESA project areas are visible on the web and cited by websites of national and international academic institutions, governmental and non-governmental institutions and industry. International web citations come mainly from US institutions (government and academic).

FESA stakeholders generally reported satisfaction with the accessibility of the Sub-Activity’s outputs. External stakeholders deem FESA scientists and staff highly responsive to their requests via direct interactions (face-to-face, phone calls and emails). Interviewed and surveyed stakeholders agreed that the current practice of providing outputs in multiple formats aids users in accessing this information. Access by the provinces, territories and academia was further facilitated by their physical proximity to scientists across Forestry Centres.

Nonetheless, some areas for improvement were identified. Around the issue of usability, external interviewees indicated that information was not always available in the most appropriate format or scale (e.g., often presented in aggregate form when looking for specific provincial data, or vice versa). Similarly, information was said to be presented in highly technical language that sometimes made it too difficult for quick and easy use.

There were also areas for improvement regarding external communications or interactions. For example, due to the large size of the department it was difficult for external stakeholders to find the best point of contact within NRCan or CFS for a given topic. There is also a need for CFS to provide more regular updates, particularly in case of notifications about new research/outputs. Provincial representatives reported that their needs could be better reflected/understood by CFS to ensure the collection of adequate and consistent information. Finally, the Sub-Activity could improve its outreach to the general public both by providing access to more data on outcomes/achievements in SFM in Canada and by making data and information currently available online more user-friendly.

Question 5: To what extent have target audiences been provided with reliable and credible knowledge, information, tools and advice on determining the state of the forest ecosystems? (P1.2 & P1.3)

FESA is perceived as a reliable and credible source of science-based information by its users and the scientific community. Among all FESA project areas, Carbon and Forest Assessment outputs are highlighted as particularly credible by both the qualitative and quantitative evidence. Moreover, information and data produced by FESA is commonly used by provinces and territories to fill their own expertise gaps and to expand internal capacity.

Credible information source: FESA is perceived as a reliable and credible source of science-based information within Natural Resources Canada,^{Footnote 91} by other federal government departments^{Footnote 92} and external stakeholders/partners. CFS scientists have a recognized reputation across the national and international scientific community and are considered well-established in their fields. CFS-generated research results, methods and data are used by the international scientific community to support and advance forestry research in their regions.

The ratio of peer-reviewed publications among all FESA outputs can be used as a proxy of the reliability, credibility and authoritativeness of the information produced under the Sub-Activity. Indeed over half of FESA’s documentary outputs have been peer-reviewed and these papers were cited in peer-reviewed publications more than the Canadian and world averages in all FESA project areas except Biodiversity. Among the six project areas, citation rates are particularly high for Carbon and Forest Assessment.

The above findings were also reflected in stakeholder views. Scientists within the Carbon project area are considered world leaders in the field and are often consulted by foreign countries during international discussions. Domestically and internationally, the Carbon Budget Model developed by CFS is considered a strong and reliable tool as evidenced by the number of countries (eight as of December 2011) that chose to use it since its launch a decade ago. The other project area highlighted is Forest Assessment which is seen as a source of scientifically credible information that confers a competitive advantage for Canada.^{Footnote 93} Its information is particularly useful in international trade^{Footnote 94} as international buyers request evidence that exported forest products do not carry pests or diseases and that the forest management/exploitation procedures follow sustainable forest management practices.

Although not to be used as a direct indicator of the data’s authoritativeness, the fact that FESA outputs are being used by many stakeholders also shows that target audiences are confident in referencing FESA-produced information, according to interviewees. Stakeholders within the IMFN group also feel that the model forest approach can be considered highly evidence-based, since the concept has been tested in practice for 20 years in a number of countries in different regions around the world.

Filling the gap: Canadian jurisdictions rely to a great extent on the information produced by the Sub-Activity to complement areas where their own knowledge or expertise is limited. For example, interviewed provincial representatives indicated that CFS brings its expertise to several areas such as fire modelling and climate change analysis. Provinces and territories also rely on CFS to compile national-level data. Furthermore, a number of stakeholders from various external organizations reported that the long-term collaboration with NRCan-CFS contributed to building their internal capacity. Some examples of increased internal capacity included improved calibration and validation of internal work and increased efficiencies (on the part of provincial/territorial representatives) and increased access to graduate students, HQP, new data and perspectives (academia). CFS scientists, with whom provincial/territorial representatives already have a strong relationship, were also viewed as easily accessible, often providing the target audiences with required assistance (e.g., for the use of CFS tools).

Finally, the evaluation survey data provided indications that there may be opportunities to further increase the target audience capacity to interpret and use CFS outputs. In particular, according to component leads, target audiences of Productivity & Dynamics had less capacity to access the outputs compared to other project areas, while Carbon audiences lacked sufficient capacity to use the project area outputs.

Question 6: To what extent do Canadians and their institutions have a solid understanding, grounded in science, of how forest ecosystems respond to natural factors and human activity? (P1.4)

Target audiences are indeed using FESA outputs to increase understanding of how forest ecosystems respond to natural factors and human activity. This is particularly notable in the project areas of Carbon and Forest Assessment.

Increased understanding: All lines of evidence confirm that FESA target audiences are continually working towards an evidence-based understanding of how forest ecosystems respond to natural factors and human activity. As shown in Figure 7, close to half the survey respondents reported that their components addressed these outcomes (i.e. to a large or great extent).

The bibliometric and usage analysis confirmed the above survey findings. In particular, the web-page study of a small selection of eight peer-reviewed papers shows that these papers were primarily cited in other peer-reviewed papers and on websites (see Table 15) to demonstrate an increased understanding of how ecosystems respond to natural and human interventions. While academia was identified as the main user sector, other users of highly cited/used documents included international government organizations, non-governmental organizations, individuals, and media.

Table 15 Result of webpage usage study for eight selected peer-reviewed papers

Selected paper			Usage sector									Reason for citing
			Government			others
FESA project area	Author (Year)	Citing websites analysed	Canada, federal	Canada, provincial	international	Academic	Encyclopaedia	Individual	Industry	Media	NGO	using data	teaching material	P1.4	P1.5	P1.6	P1.7	P1.8	P1.9
Biodiversity	Namroud et al. (2008)	1	--	--	--	1	--	--	--	--	--	--	--	--	--	--	--	--	--
Biodiversity	Aukema et al. (2008)	2	--	--	--	2	--	--	--	--	--	--	--	2	--	--	--	--	--
Carbon	Kurz et al. (2008)	20	--	--	4	8	2	1	--	3	2	--	3	18	6	--	--	1	3
Carbon	Balshi et al. (2007)	1	--	--	--	1	--	--	--	--	--	--	--	1	--	--	--	--	--
Productivity; & Dynamics	Harper et al. (2005)	7	--	--	1	6	--	--	--	--	--	1	1	2	2	--	--	--	2
Productivity; & Dynamics	Barr et al. (2004)	--	--	--	--	- -	--	--	--	--	--	--	--	--	--	--	--	--	--
NFIA	Lim et al. (2003)	1	--	--	--	1	--	--	--	--	--	--	--	--	--	--	--	--	--
NFIA	Boudreau et al. (2008)	1	--	--	1	- -	--	--	--	--	--	1	--	1	--	--	--	--	--

Note: The reasons for citing were the following:
P1.4: to demonstrate an understanding of how forest ecosystems respond to natural and human activity
P1.5: to inform the management of forest ecosystems, forecast and manage forest ecosystem risks
P1.6: to inform decisions to support forest ecosystem priorities
P1.7: to demonstrate an influence on forest management policies and practices
P1.8: to demonstrate an influence on the identification and management of risks to forests, forest ecosystems and the forestry sector
P1.9: to demonstrate an influence on international decisions on forest ecosystem policies and practices
Note: The top 50 citing websites according to Google were analyzed when appropriate. For Barr et al. (2004), 26 hits were the link to a peer-reviewed document citing it (these were not considered in this part of the analysis, but covered by the bibliometric analysis), 16 hits were a version of the document itself and 8 websites were not available. For Namroud et al. (2008), only one website was analyzed as 5 were not available, 2 did not refer to the document, 10 were electronic versions of the paper itself and 32 were links to scholarly papers, which we did not consider in this part of the analysis.

Source: Computed by Science-Metrix using data extracted from the Google search engine (http://www.google.com)

Evidence from interviews/focus groups indicates that collaboration with NRCan (e.g. with provinces/territories) included forest disturbance work (i.e., natural pests, fire etc.) to understand the forest response and to identify zones for treatment. More generally, collaboration contributed to knowledge exchange and allowed for testing of new theories, or examining new research questions on how forest ecosystems respond to natural factors and human activity.

It is important to note that many interviewees did not rely on FESA information alone to conduct these activities. In these collaborations, FESA information was most often complemented by data from other governments (international, provincial, local), academia and other research institutes. An example of such complementary efforts is the Ecosystem Management Emulating Natural Disturbance (EMEND) project (Productivity & Dynamics), which contributed to improved understanding of forest ecosystems across federal and provincial governments and industry. EMEND is an initiative created in 1997 by forest experts from the industry, government and academic sectors. It is a 100-year forest experiment whose objective is to increase understanding of the environmental, economic and social impacts of different forest management approaches. Various harvesting practices and natural disturbances patterns are tested on a1000-hectare forest located in northwestern Alberta, and scientists observe, measure and compare the results obtained with the different scenarios. All the observations and results of the experiments are compiled in a master database administered by CFS and also published in peer-reviewed papers. Several FESA scientists have contributed to the research conducted and published peer-reviewed papers as part as EMEND, which have led to improvements in industry management practices.

Carbon, Forest Assessment and other project areas: Target audiences reported increased understanding particularly in the areas of Carbon and Forest Assessment. They widely use the climate modelling tools (including the Carbon Budget Model) to better understand and mitigate climate change impacts on forest ecosystems. In terms of web impact, the National Forest Carbon Monitoring, Accounting and Reporting System (NFCMARS) which includes the Carbon Budget Model, is in the top three CFS models and tools. The NFI and the Canadian Forest Ecosystem Classification were also used by many interviewees to better monitor changes in ecosystem dynamics, forest composition and land use over time.

The documents with the largest number of hits on Google were used to understand climate change impact (Carbon) and LiDAR remote sensing (Forest Assessment). In particular, CFS’ most cited peer-reviewed publication by Kurz and his collaborates,^{Footnote 95} and also the one with the largest number of hits on Google, relates to the impact of the mountain pine beetle on forest carbon (Table 15).

Some FESA grey literature and non-publication outputs in Biodiversity, Carbon and Productivity & Dynamics also allowed target audiences to increase their understanding of how forest ecosystems respond to natural factors and human activity (see Table 16). Evidence of such use was observed across federal/provincial/territorial governments, the US government, media, and academia (including as teaching materials). The most popular non-publication FESA outputs used by target audiences included the NFI, Sustainable Forest Management Criteria and Indicators and NFCMARS (which includes the CBM) (Table 17).

Table 16 Selected CFS grey literature documents, produced between 2003 and 2012, with most Google hits per FESA project area

Bibliographic information			Google hits	Average Domain Authority	Average Page Authority
FESA project area	Author (Year)	Title
Productivity & Dynamics (#1)	Natural Resources Canada (2003a)	The Canadian Model Forest Network	422	55.2	9.6
Biodiversity (#1)  Productivity & Dynamics (#2)	Lindenmayer et al. (2008)	Salvage Logging and Its Ecological Consequences	271	75.8	6.3
Biodiversity (#2) Carbon (#1)	Lemmen & Warren (2004)	Climate change impacts and adaptation: a Canadian perspective	255	76.2	17.6
Biodiversity (#3) NFIA (#1)	Cummins & Hiratsuka (2003)	Illustrated genera of rust fungi	251	78.3	2.9
Biodiversity (#4)	Majka & Klimaszewski (2008)	Biodiversity, biosystematics, and ecology of Canadian Coleoptera	228	79.7	1.3
Productivity & Dynamics (#3)	Environment Canada (2004b)	Threats to water availability in Canada	210	75.5	9.7
Biodiversity (#5) Productivity & Dynamics (#4)	Li et al. (2011)	Landscape ecology in forest management and conservation: challenges and solutions for global change	199	75.2	4.9
Productivity & Dynamics (#5)	Gauthier et al. (2008)	Aménagement écosystémique en forêt boréale	187	65.6	9.0
NFIA (#2)	Wulder & Franklin (2003)	Remote sensing of forest environments: Concepts and case studies	173	77.7	3.8
Productivity & Dynamics (#6)	Bhatti et al. (2006)	Climate change and managed ecosystems	168	77.2	3.4
Productivity & Dynamics (#7) NFIA (#3)	Laurentian Forestry Centre (2011)	Laurentian Forestry Centre, Canadian Forest Service	161	79.0	5.9
Biodiversity (#6) Productivity & Dynamics (#8)	Reynolds et al. (2007)	Sustainable Forestry: From Monitoring and Modelling to Knowledge Management and Policy Science	160	71.7	3.4
NFIA (#4)	Köhl et al. (2006)	Sampling methods, remote sensing, and GIS multiresource forest inventory	156	74.6	3.0
Carbon (#2)	Environment Canada (2004a)	Canada's greenhouse gas inventory, 1990-2002	73	70.5	9.3
Carbon (#3) Productivity & Dynamics (#12)	Johnston et al. (2006)	Adapting forest management to the impacts of climate change in Canada	68	62.6	2.6
Carbon (#4) Productivity & Dynamics (#24)	Paré et al. (2012)	In Brief from the Canadian Forest Service, Laurentian Forestry Centre	55	84.0	7.3

Note: The number of documents per project area was identified using the Jenks natural breaks classification method to determine a meaningful cut-off value based on the number of Google hits. Six documents were selected for the Biodiversity, four for the Carbon, eight for the Productivity & Dynamics and four for the Forest Assessment project area. The project area for which the particular document was selected, is highlighted in bold, the ranking position per area is indicated in parentheses. Average Domain Authority and Average Page Authority refer to the arithmetic mean of Domain and Page authority values of all webpages mentioning the document's title.

Source: Computed by Science-Metrix using CFS publication database and data extracted from Open Site Explorer (http://www.opensiteexplorer.org/) and the Google search engine (http://www.google.com).

Table 17 Webmetrics results for selected FESA non-publication outputs

FESA project area	Acronym	Title	Google hits	Average Domain Authority	Average Page Authority
NFIA	NFI	Canada's National Forest Inventory	260	77.5	7.6
Productivity & Dynamics	C&I-SFM	Sustainable forest management criteria and indicators	200	75.5	9.0
Carbon	NFCMARS	National Forest Carbon Monitoring, Accounting and Reporting System	120	73.0	7.9
Biodiversity	CAFGRIS	Canadian Forest Genetic Resources Information System	40	66.6	1.6
Carbon	CanFIRE	Canadian Fire Effects Model	28	66.2	7.8
Biodiversity	CONFORGEN	Conservation of Forest Genetic Resources in Canada	28	55.8	1.4
NFIA	NFD	Canada's National Forestry Database	11	64.1	8.0
Productivity & Dynamics	HRB-EBM Framework	Ecosystems-based Management (EBM) in the Humber River Basin (HRB)	1	63.4	5.0

Note: Includes two selected non-publication items per FESA project area. Average Domain Authority and Average Page Authority refer to the mean of these values of all webpages mentioning the item’s title on the web.

Source: Computed by Science-Metrix using data extracted from Open Site Explorer (http://www.opensiteexplorer.org/) and the Google search engine (http://www.google.com).

Question 7: To what extent do domestic and international governmental organizations use reliable, credible information to identify, understand, forecast and manage forest ecosystem risks? (P1.5)

While target audiences are aware of risks, due to limited information, the evaluation was unable to clearly determine the overall extent to which they actually use FESA outputs to forecast and manage forest ecosystem risks. However, the existing evidence suggests that FESA does moderately contribute to some risk management activities.

Overall, limited information for this question was identified across the various lines of evidence. Some interviewees across stakeholder groups reported that FESA contributed to identifying and managing risks related to:

• Climate change impacts: e.g. managing of carbon flux over time using the CBM as a risk assessment tool, and understanding the impact of harvesting practices linked to climate change mitigation
• Natural disturbances: e.g. linked with the Forest Disturbances Sub Activity, this includes examining invasive species management strategies, prediction models for the expansion of the mountain pine beetle and risks related to fire hazards
• Impacts of forest management on biodiversity: e.g. groups studying these impacts use model forests to identify and understand risks related to species at risk

Fifty-four percent of survey respondents indicated that the outputs provided by their component(s) increased the awareness of their target audiences regarding forest ecosystem risks to a large or great extent (Figure 7). Specifically, all respondents in Productivity & Dynamics and 71% of respondents in Biodiversity declared their components were addressing this outcome to a large or great extent. Significantly fewer respondents in other components (30% to 18%) engaged in this outcome to such extent.

While the respondents indicated that their component outputs were geared toward addressing forest ecosystem risks, only 23% of survey respondents believed that target audiences used FESA outputs to manage forest ecosystems and to forecast and manage forest ecosystem risks to a large or great extent (Figure 8). Given that this was a survey of FESA component leads, ability to comment on actual external use may be limited.

Additionally, the documentary evidence suggests that the risks to forest systems are understood in Canada as the forest management legislation is considerably stringent and provides some indication of use of information to manage risks. ^{Footnote 96} Indeed, the Pew Environment Group suggests that Canada’s boreal forest is more protected than any other intact forest ecosystem in the world.^{Footnote 97} Further, the use of the Canadian Model Forest Concept and of abundant documentation prepared by the CMFN and the IMFN Secretariat resulted in highly positive reports on sustainable forest management and contributed to risk management practices in the newly established model forests in regions around the world.^{Footnote 98} Similarly, the Carbon Budget Model together with the Canadian Forest Fire Danger Rating System contribute to managing forest ecosystem risks such as fire potential and fire management activities.^{Footnote 99}

Lastly, bibliometrics and usage analysis provided evidence of use of some FESA outputs to contribute to this outcome, especially non-publication outputs such as tools, databases, frameworks, and models. In particular, seven of the eight tools and frameworks that made an impact on the web were used in the context of forest ecosystem risks management ( Table 18).

Table 18 Result of webpage usage study for eight selected non-publication items

Selected paper			Usage sector									Reason for citing
			Government			others
FESA project area	Acronym	Citing websites analysed	Canada, federal	Canada, provincial	international	Academic	Encyclopaedia	Individual	Industry	Media	NGO	using data	teaching material	P1.4		P1.5	P1.6	P1.7	P1.8	P1.9
Biodiversity	CAFGRIS	12	8	4	--	3	--	--	--	--	--	6	1	5		5	6	6	--	--
Biodiversity	CONFORGEN	7	7	7	--	--	--	--	--	--	7	--	--		7	7	7	7	--	--
Carbon	NRCMARS	24	15	--	2	3	1	--	2	1	--	9	--		13	6	1	9	--	4
Carbon	CanFRE	9	7	1	--	1	--	--	--	--	--	7	--		2	7	--	1	2	--
Productivity & Dynamics	S&I-SFM	2	2	--	--	--	--	--	--	--	--	--	--		2	2	--	2	--	--
Productivity & Dynamics	HRB-EBM	--	--	--	--	--	--	--	--	--	--	--	--		--	--	--	--	--	--
Forest Assessment	NFI	24	11	3	--	9	--	1	--	--	2	18	--		1	13	--	5	--	4
Forest Assessment	NFD	4	1	2	--	--	1	1	--	--	--	3	--		--	3	--	--	--	--

Note: The reasons for citing were the following:

P1.4: to demonstrate an understanding of how forest ecosystems respond to natural and human activity
P1.5: to inform the management of forest ecosystems, forecast and manage forest ecosystem risks
P1.6: to inform decisions to support forest ecosystem priorities
P1.7: to demonstrate an influence on forest management policies and practices
P1.8: to demonstrate an influence on the identification and management of risks to forests, forest ecosystems and the forestry sector
P1.9: to demonstrate an influence on international decisions on forest ecosystem policies and practices

Note: The top 50 citing websites according to Google were analyzed when appropriate. For HRB-EBM, Framework we found 1 hit but it actually referred to the project website itself. For C&I-SFM, there were only 2 out of the 50 websites that could be analyzed, because 3 were not available and 45 did not refer to the project because of Google search limitations).

Source: Computed by Science-Metrix using data extracted from the Google search engine (http://www.google.com).

Question 8: To what extent are Canadians and their institutions equipped to engage in forest ecosystem decision-making processes that continuously improve forest management policies and practices? (P1.7)

Multiple examples indicate that FESA’s outputs were used to provide scientific evidence to support domestic policy-making and improved forest management practices. Indirectly, FESA activities have also contributed to decision-making processes related to improved forest management.

Policies: There is clear evidence to indicate high uptake of CFS-produced scientific information on SFM among provinces and territories. Recent years have seen a number of provinces and territories introducing new legislation, notably in the area of climate change adaptation and GHG accounting. Some illustrative examples include:

• the BC Net Zero deforestation policy and the province’s decision to include potential forest carbon offsets within their policy framework;^{Footnote 100}
• changes in Ontario standards related to tree-cutting practices on shorelines; and
• an indirect contribution of FESA work to the new Forest Regime in the province of Quebec, especially in the area of caribou management.

Practices: The evaluation also found that provinces and territories and the forest industry have improved their practices with respect to forest management. Example of such improvements include the EMEND project, fire management practices, harvesting/cutting practices, NFI techniques and standards, and inclusion of CCFM C&I in forest management. CFS also frequently contributes expertise to advisory committees responsible for boreal forest management or certification licensing practices.^{Footnote 101}

Decision-making processes: The evaluation survey gauged use of FESA outputs for decision-making to improve forest management policies and practices. Use was generally considered moderate for both governmental and non-governmental organizations (see Figure 10 and Figure 11).

Further, there may be some indirect links between FESA activities and overall improved decision-making processes. For example, the State of the Forest reports shows that the management of Canada’s forests has been improving over the last two decades (i.e. decreased deforestation, reduced GHG emissions, decreased energy use and increased certification of forest third-parties).^{Footnote 102} Similarly, six of the eight FESA tools, frameworks and models selected for study in the webpage usage analysis were cited as having influenced decision-making processes to improve forest management policies and practices ( Table 18 above).

Question 9: To what extent do Government of Canada institutions use effective, defensible, science-based decision-making to support forest ecosystem priorities and to meet their national and international obligations? (P1.6)

The evaluation found some evidence that, at the federal level, forest ecosystem priorities are supported by effective, defensible and science-based decision-making informed by FESA outputs. FESA also contributes to science-based decision-making to meet international obligations, specifically by informing Canada’s position in international negotiations and by fulfilling various reporting requirements.

Context: Evidence supporting this finding must first be placed in context by identifying the main forest ecosystem priorities. Here, a number of stakeholders (both internal and external) named the following: 1) Increasing understanding of climate change impact on forests and implementing climate change mitigation and adaptation strategies; and 2) Promoting sustainable forest management practices to ensure that the forest industry remains competitive in the domestic and international markets.

Support for priorities: Decision-making within NRCan on these and other priorities is informed by FESA-generated tools and information to a great or a large extent, according to almost half of survey respondents (48%). Bibliometric analyses provide examples of such use among grey literature and non-publication outputs related to Biodiversity, particularly the Canadian Forest Genetic Resources Information System (CAFGRIS), Conservation of Forest Genetic Resources in Canada (CONFORGEN) and the National Forest Carbon Monitoring, Accounting and Reporting System (NFCMARS).

Nevertheless, some FESA interviewees noted that decisions are primarily driven by policy priorities rather than science priorities, although there have been efforts in the past few years to increase science-policy integration within CFS. It was noted that while FESA outputs certainly contribute to defensible and rigorous decision-making, they are only one of several elements considered when making a science-based decision.

Specifically within the Forest Assessment project area, it is noteworthy that the C&I component is geared toward the identification of gaps in knowledge and research priorities specifically to inform the Government of Canada position.^{Footnote 103}

International obligations: Survey, documentary and interview evidence demonstrate that the FESA Sub-Activity contributes to Canada meeting international obligations. Forty percent of survey respondents considered the outputs of their component useful for this purpose (less than 15% did not).

More specifically, FESA activities inform and assist the federal government with meeting its climate change-related international commitments under the United Nations Framework Convention on Climate Change (UNFCCC), Greenhouse Gas (GHG) Inventory and in the past, the Kyoto Protocol.^{Footnote 104} NFI-generated data are used in the production of forest related GHG estimates using NFCMARS and the Carbon Budget Model (in the Carbon project) which are then provided to Environment Canada to incorporate in its annual GHG National Inventory Report to the UNFCCC.^{Footnote 105} Similarly, the Carbon project provides forest carbon/GHG information (including the Carbon Budget Model) to Environment Canada to incorporate in the same annual reporting to the UN,^{Footnote 106} while the information from Canada’s GHG National Inventory Report was used in the past to support reporting under the Kyoto Protocol. It was observed that both the IMFN and the Forest Assessment project areas contributed to Canada’s ability to meet its international reporting obligations.^{Footnote 107}

Finally, FESA also contributes to fulfil its international engagements regarding the state of the forest. These engagements include work for the Food and Agriculture Organization’s (FAO) State of the World’s Forests study and the North American Forestry commission eco-region map.^{Footnote 108}

Question 10: To what extent does Canada influence international decisions on forest ecosystems policies and practices and meet its treaty obligations? (P1.9)

FESA’s work was found to influence international decision-making, policies and practices, as well as to aid Canada in meeting its own international treaty obligations to a good extent.

International influence: Data from interviews and focus groups provided numerous examples of the Sub-Activity influencing international decision-making, policies and practices. One of the most frequently mentioned examples was the Model Forest concept, which was recognized as having clearly influenced sustainable forest management practices in countries where model forests have been adopted. Canada was also reported to participate in international climate change discussions and to provide input towards the development of sustainable wood product certifications and assessment methods of sustainable forest management. In fact, the FESA/CFS input was used to determine how to assess sustainable forest management for FAO’s global assessment exercise to be conducted in 2015.

FESA outputs were further cited in policy contexts by international governments and organizations, particularly in the US, but also in Europe and the Middle East. Indeed, the bibliometric analysis found that some Canadian forest management publications^{Footnote 109} produced under the FESA Sub-Activity had influence on international (US) forest ecosystem policies and practices. For example, Harper et al. (2005) was cited by the US Environmental Protection Agency and by a proposal from Columbia University to improve ecological and human health in New York City. Moreover, the NASA Earth Observatory cited the publication by Kurz et al. (2008). Citations of Kurz et al. (2008) were also found in a research project description by the US Forest Service and a testimony by the Associate Director for Natural Resource Stewardship and Science in the National Park Service Herbert C. Frost before the US House Natural Resources Committee. In addition, two international government websites mentioned NFCMARS in the context of UNFCCC and international negotiations on greenhouse gas accounting. Indeed as seen throughout this evaluation report, the CBM tool (part of the NFCMARS) has been widely adopted and used in other countries.

Canada is also pioneering international agreements on sustainable forest management such as the Montreal Process international indicators, which are now included in NRCan-CFS’ annual State of the Forest reports. Following Canada’s adoption of these indicators, the US Forest Service is currently examining the practice for implementation for its own reporting. Overall, it was found that the US Forest Service is emulating Canada’s State of the Forest report.

Overall, Canada has been successful in meeting its international treaty obligations, as previously outlined under Question P1.6. It reports regularly under several mechanisms, including the UNFCCC, the FAO State of the World’s Forests report, and the Convention on Biological Diversity. The contribution of FESA to the fulfillment of international treaties lies, for example, in the use of the Carbon Budget Model (including NFI as a source of data for the model) to provide forest-related estimates for inclusion in the GHG National Inventory Report for the UNFCCC. As well the contribution of FESA is shown in the use of NFI data in the National Report to the Convention on Biological Diversity.

Question 11: Have there been any unintended outcomes (positive or negative)? (P2.1)

New collaborations and spin-off projects were found to be the positive, unintended outcomes of the FESA Sub-Activity.

Based on the survey results, 85% of component leads considered that a new collaboration or partnership unexpectedly resulted from the activities carried out under the Sub-Activity. This finding was further corroborated in interviews and focus groups, as both internal and external stakeholders reported that collaborative work with CFS contributed to increased networking or linkages with groups and/or researchers that were not in the initial collaboration. The bibliometric collaboration network analysis (based on co-authorship) demonstrates that CFS is the most well-connected institution within the national forestry network as a whole and in the four project areas examined.

Related to collaborations and networking, survey respondents indicated that FESA contributed to new or greater opportunities to disseminate knowledge information or tools (65%). Survey respondents highlighted spin-off projects (57%) as an important unintended outcome of activities carried out under their respective components. This was exemplified in the interviews by the emergence of the Land Reclamation component that was not originally planned under the Sub-Activity and was actually a result of other research carried out as part of the forest productivity project.

Other positive outcomes mentioned by internal as well as external interviewees included improved international reach that exceeded initial expectations (most notably CBM, IMFN, and EMEND), improved communications (e.g. within EC; increased networking with non-governmental stakeholders), new research areas and access to new or improved products, services or technologies.

4.2.2 Demonstration of Efficiency and Economy (Issue 5)

Question 12: What are the internal and external factors that have facilitated or hindered the achievement of intended outcomes? (P3)

The main factors that facilitated the achievement of intended outcomes include collaborations and partnerships with external stakeholders and external leveraging. The main barriers identified were insufficient resources and various issues relating to communication and coordination with other actors. This posed a challenge to supporting access and use of FESA information by external stakeholders.

Facilitators: Collaboration and partnerships with external stakeholders were found to be an important factor that facilitated the achievement of the FESA Sub-Activity intended outcomes. The diverse linkages made under the Sub-Activity include interactions, relationship-building and exchanges with a wide range of forest stakeholders. In this context, mutually beneficial relationships between CFS and various stakeholders – particularly provinces/territories, academic researchers and industry representatives – were reported in interviews and focus groups as having facilitated knowledge/resource exchange and communication necessary to make progress towards stated goals. Importantly, CFS was also seen by both internal (FESA) and external stakeholders as playing a key “convenor” role to help facilitate effective collaborations between the various stakeholders involved in producing and using forest science and information across Canada.

Similar evidence was seen in the survey responses from the perspective of FESA staff. Indeed, high proportions of FESA component leads identified partnerships with provinces/territories (79%) and with academia (70%) as significant facilitators of the achievement of intended outcomes (Figure 11). Partnerships with international organizations (41%) were also seen as having benefited the Sub-Activity.

Other factors further supported these mutually beneficial collaborations. For example, many interviewees across groups cited the strong scientific capacity of CFS – including leading expertise in a wide range of forest ecosystem topics – as a facilitator to achievement of the Sub-Activity’s outcomes. Half of the survey respondents also identified the availability of FESA personnel with required skills/expertise as an important facilitating factor (Figure 11). Many external interviewees and focus group participants also perceived FESA staff as open, responsive and highly engaged in their work.

A second key facilitating factor, external leveraging (both funding and in-kind contributions) from OGDs, Provinces/territories and academia, was critical to the achievement of FESA outcomes. Interviewees and survey respondents stated that without external support, FESA projects would not likely occur. Indeed, three quarters (75%) of survey respondents identified funding from sources outside NRCan as a major facilitator (Figure 11). In addition, none of the survey respondents indicated that the components for which they were responsible would have proceeded in full without additional monetary support from external sources; about 50% indicated the work would have proceeded with a more limited scope and over a longer timeframe. The importance of external sources was further corroborated by the FESA financial data, which indicated that support from external resources (funding and in-kind) often made up the difference between the requested and allocated internal resources in several project areas.

Note that some interview data and documentary evidence revealed some facilitating factors specific to the IMFN. Particularly, the success of this project area was seen to be due in large part to its stable and mature governance structure. It also benefitted from the increase in the number of full-time equivalent staff following the transfer of the IMFN Secretariat from IDRC to CFS.^{Footnote 110}

Challenges: In terms of hindering factors, the interview and survey data indicated that the activities of FESA were constrained by declining human and financial resources exemplified by budget cuts, retirements, staff departures and insufficient replacements. Indeed, close to half (46%) of survey respondents identified NRCan funding, or lack thereof, as a factor that greatly inhibited the achievement of intended outcomes (Figure 11). Similar consensus views were provided by both internal and external stakeholders.

In addition, according to the documentary evidence, the economic downturn in the US market (2008/09), as well as the high exchange rate and other economic factors had a strong negative influence on the forestry sector during the period under evaluation.^{Footnote 111} As indicated by several interviewees, these unfavourable economic trends limited P/T and industry resources. This in turn limited the ability to invest in collaborative projects with CFS, which meant negative implications on external leveraging.

Challenges in achieving the intended outcomes of the Sub-Activity were also encountered with respect to coordination and dissemination of information to support stakeholder access and use. Specifically, interviewees, focus groups participants and survey respondents pointed to difficulties with obtaining consistent and timely information from provinces/territories (e.g., for the compilation of NFI). At the same time, provincial/territorial representatives noted that their data needs have not always been understood by CFS (e.g., in terms of how NFI data were presented). Further limitations that were frequently reported by interviewees including travel restrictions (both national and international) for CFS staff and other stakeholders, which hindered their ability to network with relevant audiences, to strengthen relationships (e.g., establish trust), to share ideas and to more fully disseminate FESA outputs. Dissemination of the Sub-Activity’s research results was seen to be negatively impacted by current NRCan, federal and provincial/territorial government travel policies and restrictions.

Some barriers around coordination and dissemination were also echoed by a number of survey respondents. Specifically, 21% of the component leads reported in the survey that the availability of FESA personnel, or lack thereof, with required skills inhibited the achievement of expected outcomes (compared to 50% who identified this as a facilitating factor). This may also be related to the human resources issues mentioned above. As well, a finding from the interviews and focus groups was that the ability to directly contact CFS staff in their regions was a facilitating factor, while the fact that CFS staff were not represented in every province or territory hindered the ability for some provinces and territories to access CFS expertise relating to FESA.

Overall, both interviewees and survey respondents pointed to potential avenues to improve the communication between FESA staff and their target audiences. These suggestions include an increased focus on dissemination of outputs/tools (which may involve greater support from NRCan’s corporate communication services), holding workshops with external stakeholders to improve coordination of activities, and more upfront communication to better understand the needs of target audiences and clarify CFS role with regard to these needs.

In the case of the Forest Assessment project area, improved coordination with provinces/territories and OGDs could help address specific issues that hindered the effectiveness and efficiency of this project area. These issues include lowering operating costs, improving leveraging of resources (e.g., under the Memorandum of Understanding for the NFI), and supporting the re-measurement and assessment process (data collection) for the NFI to address ongoing delays in this regard.

Finally, some challenges were noted with regard to internal coordination. Specifically, CFS management approach (i.e., matrix model) was found to inhibit the achievement of expected outcomes by 25% of component leads (compared to 18% who found this approach to be a facilitator). In contrast NRCan senior management and FESA project coordinators and peer directors viewed this management approach – and the overarching FESA governance structure – as a positive practice. In particular, they identified a number of related improvements over the evaluation period (e.g., innovation management initiative). Nonetheless, these internal interviewees acknowledged that this management approach did pose some challenges, including the additional administrative burden (e.g., multiple meetings, use of ProMIS), priority-setting processes (e.g., matching between science priorities and policy priorities) and internal communication. These findings are further discussed in the next section.

Question 13: Are the project areas the most economic and efficient means of making progress towards the intended outcomes? (P4)

An analysis of the financial data indicates an overall decrease of internal FESA resources over the evaluation period. Similarly, the leveraging of external funding and in-kind resources has also decreased. This may significantly affect FESA work as external leveraging represents a substantial share of total FESA resources.

Currently, there are no systematic processes for performance data collection/progress reporting, meaning that such information is not readily available to help inform decision-making. While greater efforts towards science-policy integration are being made, improving the priority-setting process and enhancing the ProMIS system to allow for performance tracking were provided as the most common suggestions to improve the efficiency and economy of the Sub-Activity.

Finally, a best practice to maintain is the effort to bring together a wide range of stakeholders though partnerships, collaborations, workshops and joint projects. This was deemed highly effective way to share resources, expertise and to make progress towards achievement of planned outcomes.

Internal resources for program delivery: The cost of program delivery for the FESA Sub-Activity was $101.5 million from 2007-08 to 2011-12, including $84.4 of internal expenditures (i.e., actual NRCan funding, including A-base, C-base, G&C and transfer payments), $5.7 million in external financial resources (actual) and $11.4 million in-kind support (planned). Most expenditures are used to support salaries (over 75%), followed by O&M (about 20%), although this varies across project areas (e.g., 30% of Forest Assessment expenditures are for O&M). Details are provided by project area in Table 9 and Table 10 (Section 2.5).

Examining trends in resources over time, internal FESA resources have decreased gradually over the evaluation period, from $18.7 million in 2007-08 to $14.3 million in 2011-12 (with the exception of a second peak of $18.1 in 2010-11, which appears largely due to the IMFN). In terms of individual project areas, only Forest Assessment appears to have maintained its internal resources over the period while Carbon experienced a slight increase; all other project areas have generally seen a decline in their internal resources (Figure 12). Note that salaries and fixed costs have increased during this period (because of inflation and increased pay rates), such that the actual purchasing power of a dollar has decreased over time.

Project areas also differ in terms of the extent to which they rely on C-base funding: nearly 70% of IMFN’s internal resources were from limited duration C-base funds (including G&C, excluding the African Model Forest Initiative) over the evaluation period, while this proportion was 36% for Carbon and 15% for Land Reclamation. Overall, 17% of FESA’s internal resources were from C-base funds. C-base funding appeared to vary considerably from year to year, such that no clear trends were observed over the evaluation period (data not shown).

Moreover, there was an overall decline in the percentage of requested A-base and C-base funds allocated for all project areas from 2008-09 to 2011-12 (data are not available for 2007-08): for FESA as a whole, this percentage decreased from 71% to 45%, with an average of 46% over the evaluation period. As shown in Figure 14, the Carbon project area was allocated almost 100% of requested funds in both 2008-09 and 2009-10. However, for these two years, funding allocation percentages were much lower for the Forest Assessment, Biodiversity and Productivity & Dynamics project areas. These were allocated between 60% and 70% of requested resources in 2008-09 and between 37% and 47% of requested resources in 2009-10. Furthermore, by 2011-12, no project area was allocated more than 58% of requested resources, with four project areas (Biodiversity, IMFN, Productivity & Dynamics, and Land Reclamation) below 40%. Although, IMFN received only 4% of the requested funding in 2010-11, it should be noted that that year’s request ($4.9 million) greatly exceeded the amounts it requested or was allocated in the other two years for which data is available ($0.2-0.6 million).

External resources for program delivery: With regard to leveraging from external sources, a similar trend was observed across project areas. Overall, 6% of FESA resources are obtained from external financial sources, while 11% was planned to be obtained from in-kind sources, together representing 17% of the total FESA resources. Forest Assessment leveraged the highest amount of both external financial resources ($3.2 million, 10% of its total resources) and in-kind resources ($6.3 million, 19% of its total resources), although IMFN leveraged the highest proportion of financial resources in relation to its total resources (12%, $0.9 million).

The amounts of external financial resources (as well as the share of the total this represents) have decreased over the period, for FESA overall (from $1.7 million in 2007-08 to $0.7 million in 2011-12), as well as for most project areas (Figure 14). This trend is also reflected in the percentage or ratios of external financial resources in relation to total resources (data not shown), suggesting that external funding has been decreasing to a greater extent than internal funding.^{Footnote 112} As noted under the previous question, this decline may be related to the economic downturn. Note that no detailed data was available with regard to the sources from which external resources were obtained, but these sources are understood from interviewees to include OGDs, Provinces/territories, academia, as well as industry.

As noted in the previous section, external leveraging was seen as a key facilitating factor in the achievement of FESA outcomes. This leveraging is important not only because it represents a substantial share of total FESA resources, but also because it is appears to compensate to some extent for the difference between the amounts requested and allocated from internal resources. Looking at leveraging of external funds (financial only), this typically represented about 40% of the difference between requested and allocated funds (up to 86% for Forest Assessment), in some cases, this even exceeded this difference. If in-kind resources are considered as well, then the external leveraging (financial and in-kind) often more than made up the difference.

Implementation as planned: Evidence of the extent to which the project areas were implemented as planned was generally limited to interview data; the database used to manage FESA activities – ProMIS – is a planning tool and does not track the outputs or level of achievement of specific activities (see related discussion on performance measurement data, below). Interviewees largely confirmed and fleshed out the implications of the trends noted in the financial data discussed above. In particular, Land Reclamation, Forest Assessment and IMFN activities were seen to be particularly affected by the lack of funding and associated limitations in human resources. For example, work under Land Reclamation was delayed because of insufficient available scientific resources to carry them out. This is related to the challenge that CFS is experiencing around identifying and clarifying its role in this project area, since many other players are already active in Land Reclamation research. In the case of Forest Assessment, NFI’s re-measurement cycle has reportedly been delayed due to lack of resources. IMFN has experienced a recent decline in funding since its move to NRCan from the IDRC. Moreover, scientific activities within the Biodiversity and Productivity & Dynamics project areas were also said to have been affected by the declining level of resources; note that these two project areas consistently received less than 60% of the requested resources (Figure 13).

Interviewees indicated that all of these findings (declining internal and external resources, declining allocations vs. requested amounts, reliance on C-base funding) also have implications for the planning of FESA activities, both in the short and longer term. Stakeholders consulted via the interviews, focus groups and survey also provided a number of suggestions to improve the effectiveness, efficiency and economy of the FESA Sub-Activity given the current context. Existing practices to support FESA’s efficiency and economy, as well as suggestions for ongoing improvement, are discussed in the paragraphs below.

Opportunities to increase efficiency and economy: Practices to increase the efficiency and economy of the FESA Sub-Activity have been identified by CFS across all project areas, and several have been implemented. For example, interviewees noted that NRCan has implemented new management procedures that ensure that project inputs (i.e., funding), outputs and CFS/department/Government of Canada outcomes are aligned. These procedures rely on the Sub-Activity governance structure (including the input of project directors and peer directors). ProMIS (see below) is also being used to support these new procedures.

There is also documentary evidence of processes that CFS has put in place to support the implementation, tracking and management of this Sub-Activity and its different project areas. In particular, all components are recorded in ProMIS, a project planning system that stores a number of variables (e.g., component description, component lead, budget requested and allocated for every year).^{Footnote 113} However, as noted previously, ProMIS is used predominantly for planning of activities and intended resources rather than project monitoring and management. Furthermore, the ProMIS system was seen by component leads as inefficient for setting science priorities or strategically allocating resources to individual components, given the additional administrative burden involved. Interviews with CFS management-level staff confirmed this burden was a concern and efforts have been made to reduce it (e.g., by reducing the number of components). However, these interviewees generally agreed that ProMIS would be a rather inefficient tool for tracking results (i.e., outputs, outcomes) and some suggested that other sources of information should be used for this purpose.

Hence, there are currently no systematic performance data collection processes within the Sub-Activity. For example, the Sub-Activity does not produce regular quarterly or annual progress reports for its activities, with the exception of those supported from C-base funding and IMFN.^{Footnote 114} Instead, according to interviewees from CFS, performance data is tracked through ad hoc mechanisms such as the high level reporting to TBS or NRCan, occasional evaluations and audits, individual (staff) assessments, and requests from senior management. Additionally, some long-term outcomes are tracked via the State of Canada’s Forests report.

This lack of systematic performance data collection means that such information is not readily available to help inform decision-makers about the performance of the Sub-Activity. Thus, although there are management procedures in place to help support the achievement of FESA outcomes in an economical and efficient manner, performance measurement does not appear to be contributing in a consistent and rigorous way to decision-making. With regard to the evaluation of efficiency and economy, although resource utilization data is available for each project area, the low availability of program data on the specific outputs and outcomes that have been achieved by each project area (not only those that were planned), means that a comprehensive analysis of efficiency and economy^{Footnote 115} on the basis of documentary evidence is limited.

As another recent practice that contributes to improving the efficiency and economy of the FESA Sub-Activity, CFS is also moving towards a vertical and horizontal integration of knowledge to ensure that scientists and policy-makers increasingly work together. These current efforts at greater science-policy integration are already seen by some internal interviewees to contribute not only to the greater alignment of FESA’s scientific activities with policy needs (e.g., as in the case of the new Ecosystems Integrity project area), but also contribute to a greater understanding and use of the knowledge produced by FESA within the federal context. Senior management representatives in particular noted that significant efforts have gone into improving the priority-setting process through greater integration of science and policy sectors within CFS. An ongoing need was identified to continue these efforts in order to further strengthen science-policy integration.

However, many of the comments provided by survey respondents regarding the improvement of efficiency and economy were related to priority-setting in the context of science-policy integration. In particular, it was reported that changes in policy priorities could have negative impacts on FESA’s scientific production, especially if these changes are too frequent. As further explained by some interviewees, most scientific activities operate on a longer timeframe than many policy priorities, such that trying to change the focus of scientific activities in response to policy needs may result in wasted efforts and lessen the quality of the scientific output. While internal interviewees were generally satisfied with the new priority-setting process in place, some noted that the match between science and policy priorities is not always optimal.

Best practices: Finally, several best practices developed or pursued under the Sub-Activity have been identified, particularly in the documentary evidence and interviews. These include practices that interviewees suggested should be maintained or considered by other programs to improve their effectiveness and efficiency.

Considering limited (and declining) resources available, internal stakeholders reported that NRCan implemented improved resource allocation procedures that were focused on ensuring alignment between project inputs/outputs/outcomes and policy priorities. Looking at specific project areas, IMFN as a whole was considered within NRCan as a best practice. In particular, it was seen to bring various stakeholders together, promote cost-effective sustainable forest management, combine forest ecosystem science and social sciences, and helped link science to policy and policy to practice.^{Footnote 116} The IMFN was also deemed to be maturing at a time when new challenges such as industry transformation, climate change, community sustainability were becoming more urgent at the global level. Moreover, the increasing intensity of resource demand has resulted in a significant increase in integrated landscape management globally has put the IMFN and its methodology at the center of a concerted effort to manage competing resource demands at scale. In the case of Forest Assessment, collaborative efforts facilitated by CFS around data collection processes include the ongoing search or innovative technologies to improve national reporting on forests.

Other efforts in bringing together stakeholders, such as through partnerships, collaborations, workshops and projects (e.g., EMEND) were deemed highly effective and beneficial for the achievement of planned outcomes, including through the sharing of resources and expertise. However, it was noted by some survey and interview participants that resources to conduct workshops would need to be augmented in order to achieve an optimal level of communication between the scientists and the target audiences. Also related to collaborative practices, the development of the innovative framework of C&I was highlighted as a best practice, as it helped Canada – and other countries that adopted the tool – support the implementation of SFM practices.^{Footnote 117}

5.0 Conclusions and Recommendations

Relevance:

Overall, the evaluation found that there is a continued need for the program Sub-Activity as FESA continues to be highly relevant to address several environmental and economic needs.

FESA also provides specialized expertise to fill a gap in areas not covered by other players by virtue of CFS’ unique and long-standing scientific tradition. FESA is also a national information convenor and facilitates market access.

There a legitimate, appropriate and necessary role for the federal government and NRCan-CFS in forest ecosystem science. Indeed the activities undertaken by the FESA Sub-Activity focus on the production of scientific knowledge and advice for sustainable forest management and the coordination of information at the national level. This convening role is key and the need for it continues.

The evaluation further concludes that the federal government is also uniquely positioned to represent Canada’s interests globally and to address international obligations. However, while the role and ongoing need for the federal government involvement was identified for all project areas, CFS should further examine and clarify its role for the Land Reclamation project area and the IMFN project area, particularly in the context of the given federal priorities, other stakeholders and available resources (see Recommendation 1).

Moreover, regarding the creation of the Ecosystem Integrity project area, CFS should further clarify whether the biodiversity capacity (developed over the last five years) has been adequately integrated within the new Ecosystem Integrity project. The expertise gained through the Biodiversity project area is relevant to Canada’s interests globally and to address international obligations.

Recommendation 1: It is recommended that NRCan further examine the relevance of FESA’s role in the International Model Forest Network and Land Reclamation. CFS should explore opportunities to position IMFN as a platform for disseminating CFS scientific research and knowledge on sustainable forest management. CFS should also clearly identify the objectives and key research questions that Land Reclamation needs to address.

The IMFN represents a unique value for Canada in terms of international outreach and leadership in forest management practices. Given the challenges of hosting IMFN in another organization, CFS should further engage within the department to explore opportunities to position IMFN as a platform for CFS scientific research, knowledge mobilization on sustainable management of forest-based landscapes and for other relevant actionable issues.

Land Reclamation as a new project area must define its role relative to other already active players. CFS must clearly identify its objectives and the key research questions that this project area needs to address. Importantly, the funding level should match the stakeholder expectations, especially given the federal priorities for this project area. Provinces and industry are expecting CFS to engage multiple stakeholders given their national presence and reach.

Recommendation 2: Given the partial integration of the Biodiversity project area into the Ecosystem Integrity project area, it is recommended that NRCan further examine any risks associated with the reallocation and/or reduction of biodiversity expertise for CFS and the federal government.

Performance:

Science-policy integration

Most of FESA project areas have been successful in producing outputs and immediate outcomes, which have been accessed and used by target audiences. However, the initiation of the Land Reclamation project area has not been optimal. Lack of resources, and a clear mandate coupled with the complexity of the issues faced by the sector impeded its initial implementation.

CFS was also found to be a reliable and credible source of science-based information by its users and the scientific community. By supplying scientific information and expertise, the Sub-Activity also plays an important role in complementing the capacity of provinces and territories.

The CFS has increasingly focused its available resources and science towards policy priorities over the evaluation period. Such changes are likely to have an effect on capacity, expertise, and services historically made available to traditional end-users, mainly provinces but also the academic and private sectors. CFS forestry centres have established long-lasting relationships with users and partners in regions. It is likely that these relationships will be impacted by this greater focus for science integration with current and changing federal policy priorities. The evaluation finds that to date, such changes have been effectively implemented internally without being well-communicated to users. Also important, the evaluation found that internal stakeholders highlighted a need for ongoing and improved information flow between scientists and policy staff in order to overcome alignment challenges.

Recommendation 3: Given the evolution of CFS science towards meeting federal policy priorities, it is recommended that NRCan clearly communicate its current and longer term plans to provinces (and other key users) by specifying what changes in expertise and outputs traditional users can expect. CFS should also consider enhancing dialogue between FESA scientists and CFS/NRCan policy staff to help clarify communication plans for traditional and new users.

Resources to improve interactions with direct users and partners for effective science transfer and leveraging

The evaluation found that interaction between CFS scientists and target audiences is essential for the full realization of FESA outcomes. Indeed, the outputs, expertise and services provided under this Sub-Activity are primarily accessed via personal interactions with CFS staff and collaboration on joint projects. Furthermore, external leveraging tends to compensate for declining internal resources and such funding mechanisms rely on frequent interactions with current and potential partners.

Some barriers to these interactions have been identified and the evidence indicates that more interactions would be beneficial for partnerships, as well as for the full realization of the access, interpretation and use of CFS science, expertise and tools.

Recommendation 4: Within the context of limited resources, it is recommended that NRCan enhance opportunities for its scientists to interact more frequently with target audiences, to ensure the effectiveness of the Sub-Activity and enable FESA to secure more external resources (financial and in-kind).

Resources to improve public dissemination of science plans and outputs

In addition to increased interactions between CFS scientists and FESA stakeholders that would augment the access and use of the Sub-Activity outputs, there is also a need for broader dissemination of ongoing and planned research activities and outputs of the Sub-Activity in the public domain. Some recent developments imped external dissemination, notably, reorganization of departmental communications unit, new requirements to reduce the content of web pages and reduced communication staff. According to FESA representatives, this has decreased the emphasis on science communications.

Recommendation 5: Within the context of limited resources, it is recommended that NRCan enhance the public dissemination of the science produced by the Sub-Activity. An overall lack of awareness of FESA ongoing research activities, as well as issues around accessibility of information, were observed among target audiences. NRCan and external stakeholders expressed a need to consider FESA activities in their planning but are not always up-to-date on upcoming initiatives. Better dissemination of FESA science will facilitate the communication of current and changing priorities (see Recommendation 3).

Performance tracking

It is challenging to track outcomes of science-based activities, especially for a sub-activity of national scope with such diverse project areas and target audiences. Nevertheless, lack of systematic processes for performance data collection/progress reporting, means that such information is not readily available to help inform FESA decision-making. There is an opportunity to revisit the program logic to better align the theory with ground-level objectives and activities.

This would facilitate the development of measurable indicators and a system for tracking performance data. Such efforts would benefit by being deployed in collaboration with other CFS science-based Sub-Activities and with NRCan SED for support and advice. FESA should use and adapt existing systems to generate performance and financial data. This can help identify existing data, unexploited information and critical gaps for the design of a new performance strategy.

Recommendation 6: It is recommended that NRCan continue to improve the tracking and reporting of performance and financial data to ensure that reliable information is used to manage FESA activities. There was an internal recognition that an improved performance measurement strategy will increase CFS ability to monitor its achievements. Given changing CFS/federal priorities, such a strategy will also help clarify the targeted FESA end-users and their needs. Existing mechanisms such as ProMIS, C-Base reporting, HR review and the publication database could be leveraged to create a better system along with improved tracking of actual financial data (internal and external sources).