Table of Contents
- Executive Summary
- 1.0 Introduction
- 2.0 Program Background and Profile
- 3.0 Evaluation Objective, Scope, and Methodology
- 4.0 Evaluation Findings
- 4.1 Relevance
- 4.1.1 Continued Need for the Program (Issue 1)
- Question: Did the PPGTP address demonstrable needs?
- 4.1.2 Alignment with Government Priorities (Issue 2)
- Question: Was the PPGTP consistent with government priorities and NRCan strategic objectives?
- 4.1.3 Alignment with Federal Roles and Responsibilities (Issue 3)
- Question: Was there a legitimate, appropriate and necessary role for the federal government in the Program?
- 4.1.1 Continued Need for the Program (Issue 1)
- 4.2 Performance
- 4.2.1 Achievement of Expected Outcomes (Issue 4)
- Question: To what extent have the PPGTP’s intended outcomes been achieved?
- Question: Were there any unintended (positive or negative) outcomes?
- 4.2.2 Demonstration of Efficiency and Economy (Issue 5)
- Question: Is the Program the most economic and efficient means of achieving outputs and targeted outcomes?
- Question: Is performance information being collected and used on an ongoing basis?
- Question: To what extent can the outcomes measured be attributed to the PPGTP?
- Question: Overall, are there any best practices and lessons learned that NRCan should consider in developing initiatives of similar size and scope?
- 4.2.1 Achievement of Expected Outcomes (Issue 4)
- 4.1 Relevance
- 5.0 Conclusions and Recommendations
The Pulp and Paper Green Transformation Program (PPGTP) was established by the Government of Canada (GoC) and implemented by the Canadian Forest Service (CFS) within Natural Resources Canada (NRCan). Its objective was to improve the environmental performance of Canada’s pulp and paper industry, contributing to environmental and hence commercial sustainability. The Program assisted Canadian pulp and paper companies in making capital investments that would improve facility environmental performance, by providing a $0.16 per litre credit for black liquor produced at mills between January 1, 2009, and December 31, 2009, or to a cap of $1.0 billion in total credits issued, whichever came first. The Program’s terms and conditions officially expired on March 31, 2012, the last day for firms to draw on their allocated credits.
Evaluation Scope and Objectives
The purpose of the evaluation was to assess issues and report findings relating to the relevance and performance of the PPGTP and provide recommendations as necessary. The evaluation covered the period from 2008-09 to 2011-12 and includes $975.5 million in expenditures. Given that the Program ended on March 31, 2012, the evaluation also aimed to critically assess the factors underlying the success of the Program, as well as to identify best practices and lessons learned that can be of value for other initiatives within NRCan.
The evaluation addressed the five core evaluation issues defined by the Treasury Board Secretariat in the Directive on the Evaluation Function, effective April 2009. Under relevance, the evaluation issues are: Continued need for Program; Alignment with government priorities; and Alignment with federal roles and responsibilities. Under performance, the evaluation issues are Achievement of expected outcomes; and Demonstration of efficiency and economy.
The evaluation was designed to draw on various sources of data to ensure that the combined lines of evidence resulted in an in-depth and comprehensive analysis. The four methods used to collect and analyze evidence were 1) a document file and data review; 2) stakeholder interviews (36); 3) an online survey of funded project participants (N = 115); and 4) three case studies on Program planning, management and delivery.
Continued need for Program: The Program addressed demonstrable needs in an evolving global market environment by providing access to capital financing directed towards projects that would improve environmental performance of the pulp and paper industry and thereby strengthen economic viability when access to capital was severely constricted. The evaluation found that in the years leading up to Program implementation, there was a limited ability for companies to make critical environmental upgrades due to declining industry revenues the global economic recession and other competitive pressures.
Alignment with government priorities: The PPGTP was consistent with both federal government priorities and NRCan strategic objectives. Federal priorities over the evaluation period consistently displayed a commitment to developing the forest industry, as reflected in Speeches from the Throne, the Clean Air Agenda (CAA) and the Economic Action Plan (EAP). The PPGTP was also aligned with NRCan goals as outlined in the Program Activity Architecture (PAA) (from 2009 to 2012), under Strategic Objective #2, Environmental Responsibility.
Alignment with federal roles and responsibilities: The evaluation found that the Program was strongly aligned with federal roles and responsibilities, especially for issues concerning international trade where the GoC has a clear constitutional responsibility. The federal government played a legitimate, appropriate and necessary role in protecting Canadian market interests by creating a targeted pulp and paper program that promoted the sustainability of the forest sector. Contribution Agreements (CAs) were found to be an appropriate mechanism to deliver PPGTP funds and to hold industrial recipients accountable for environmental outcomes.
Achievement of expected outcomes: Evidence demonstrated that the PPGTP achieved its intended outcomes to a great extent. All eligible pulp and paper mills (a total of 24 companies) were engaged via targeted outreach efforts, and 96% of proposals were funded (N = 102). The Program also had a large and well-organized record of outreach activities spanning its entire lifecycle, such as presentations, workshops, webinars, official press releases and/or media events, as well as dissemination of clear guidelines and information on the CFS website.
The PPGTP exceeded its energy savings and energy production targets:
- Energy savings of 8.4 million gigajoules per year (GJ/yr) were achieved, double the target of 4.2 million gigajoules.
- Energy production of approximately 2.9 million megawatt hours (MWh/yr) was achieved, about 0.8 million megawatt hours or 36% more than the target of 2.1.
As well, environmental improvements were noted for air emissions, water use and chemical waste.
The evaluation found that the PPGTP contributed to the environmental sustainability of the pulp and paper industry. For example, PPGTP projects collectively contributed to reducing the industry’s consumption of heavy oil by 135 million litres a year. This accounted for as much as 45% of the total Canadian pulp and paper industry consumption (which is about 300 million litres a year).Footnote 1 The Program also generated positive commercial impacts for participating companies as the dollars generated or saved by the participating mills would constitute a net income after four years. For the industry as a whole, the one-time investment of $1 billion would be fully recovered after four years with a total (for all projects combined) yearly return on investment (ROI) of about $320 million (or 26% of the initial investment).Footnote 2 The evaluation independently confirms the findings of the internal PPGTP Report on Results with minor differences accounted for in the timeframe of the available information.
Given that benefits from capital investments generally accrue over the long term, those improvements are expected to continue into the future. In fact, in an open-ended question, some industry interviewees at the executive level indicated that the PPGTP generated interest in completing other projects to improve environmental performance. As well, when asked to identify contributors to the longer-term environmental and commercial sustainability of their organization in another open-ended question, 7% of survey respondents named the PPGTP specifically.
The evaluation identified a number of facilitators enabling the achievement of Program outcomes, including clear political will and senior-level support; diverse in-house expertise; and an eligibility-based, 100% funding model with flexibility on location of investment. Retroactive payments were also permitted in order to reflect the economic realities of the industry and respect condensed Program timelines.
The main challenge for project execution and delivery, however, was the Program’s short timeframe for project application and implementation. The length of the project review process (including the environmental assessments) was perceived as a secondary challenge.
Two main positive unintended outcomes were observed as a result of this Program: 1) internal capacity building at CFS; and 2) relationship building between CFS and the pulp and paper industry stakeholders. Additionally, documentary and survey evidence showed signs of some ancillary benefits, such as job creation and increased competitiveness.
Demonstration of efficiency and economy: The Program efficiently managed its resources and activities. Several insightful management and operational procedures—including the use of Memoranda of Understanding (MOUs), multi-project Contribution Agreements (CAs), and a streamlined staffing strategy—contributed to effective and economic use of the given resources. Planned expenditures were in line with actual spending and program operating costs (as a percentage of actual total program funding) averaged 2.5%.Footnote 3 The operating cost ratio, staff size, and service standards for PPGTP were comparable to similar initiatives (e.g., Industry Canada’s Strategic Aerospace and Defence Initiative and Agriculture and Agri-Food Canada’s Farm Income Payment Program).
While the Program timeframe was strictly adhered to, 70% of survey respondents who answered an open-ended question noted that the limited time to implement projects resulted in increased difficulties, particularly for projects that were approved in the final years of the Program. However, by the end of the Program, timelines were met and industry and other stakeholders contributed more than $250 million to PPGTP projects, totalling $1.2 billion in direct investments.Footnote 4
External stakeholders suggested there were opportunities for administrative improvement regarding clearer reporting templates and flexibility around funding release. A lack of full-time resources or budget allocated for Program activities after March 31, 2012 meant that resources had to be diverted from elsewhere in the Department to conduct closing activities.
Performance information has been collected and used since the implementation of the Program. Notably, the Program’s performance metrics have been used to contribute to departmental performance reports and to create the Program’s Report on Results. However, the evaluation found that there was a lack of clarity surrounding how the post-Program data reported by the project proponents would be used, either for future reporting or to provide information on the longer-term outcomes and impacts of the Program.
Interview and survey data indicated that without PPGTP funding, about half of the projects were not likely to have proceeded, and the environmental benefits associated with the Program would not have been realized. Those that would have proceeded without PPGTP funding would likely have been carried out over a longer timeframe or with a reduced scope, and it is possible that the environmental benefits would not have been realized in the evaluation period.
Based on interviews, documents and case studies, the evaluation identified four lessons learned that NRCan could consider in developing initiatives of similar size and scope.
- New programs are advantaged by the flexibility to build on existing strengths and avoid outdated /old practices that create constraints.
- A combination of external factors/needs and strong political resolve may favour Program creation and implementation.
- Program delivery is facilitated by a) clearly defined roles within a committee or team-based governance structure and b) the opportunity for stakeholder consultation during the early planning stages.
- Program efficiency is clearly influenced by key management and operational decisions.
Conclusions and Recommendations
The evaluation concluded that the PPGTP was relevant as it addressed a demonstrable need for the Canadian public and the pulp and paper industry. Additionally, the PPGTP objectives were aligned with government priorities as well as with the departmental strategic objective of Environmental Responsibility. There was a clear role for the federal government in this type of programming.
The evaluation also concluded that the intended outcomes of the PPGTP were achieved, with good progress towards the longer-term environmental sustainability of the pulp and paper industry. Positive commercial impacts were also achieved. The PPGTP also clearly resulted in enhanced capacity-building and relationship-building for both internal and external stakeholders.
Based on these findings, the following five recommendations are made:
Recommendation 1: Building on the linkages established through the PPGTP, CFS should continue to develop its understanding of industry operations and maintain strong working relationships with its industry stakeholders.
Recommendation 2: NRCan (and CFS) should continue to leverage the unique elements and best practices of PPGTP design in order to capitalize on the Department’s emerging ability to effectively deliver a large, industry-focused initiative.
Recommendation 3: To leverage the utility of post-project data collection, CFS should consider establishing a formal means to compile and communicate the longer-term outcomes of the PPGTP
Recommendation 4: NRCan should consider documenting the PPGTP life-cycle costs and other implications for CFS to inform future design and risk assessment for grants and contributions programs.
Recommendation 5: CFS should consider the opportunity to capture key socio-economic data and report back to Canadians for future industry-focused Programs with a national scope and large materiality.
|Recommendation||Management Response and Action Plan||Responsible (Target Date)|
|1. Building on the linkages established through the PPGTP, CFS should continue to develop its understanding of industry operations and maintain strong working relationships with its industry stakeholders.||The CFS agrees with the recommendation and will continue to actively engage directly with industry stakeholders through the delivery of its existing policy and program related initiatives and particularly via the existing Investments in Forestry Industry Transformation program (IFIT) interactions. CFS will strive to maintain baseline capacity to support continued engagement efforts towards better understanding of industry operations and to sustain established relationships.||ADM – CFS (ongoing)|
|2. CFS should continue to leverage the unique elements and best practices of PPGTP design in order to capitalize on the Department’s emerging ability to effectively deliver a large, industry-focused initiative.||The CFS agrees with the recommendation. In 2012, the PPGTP assembled a Lessons Learned document on the development and delivery of the program. The program sought feedback on this report from different stakeholders within NRCan. The document is available for dissemination in both official languages and can be found on the NRCan Wiki (http://wiki.nrcan.gc.ca/index.php/Pulp_and_Paper_Green_Transformation_Pr....||ADM – CFS (ongoing)|
|3. To leverage the utility of post-project data collection, CFS should consider establishing a formal means to compile and communicate the longer-term outcomes of the PPGTP.||The CFS agrees with the recommendation and proposes to communicate the post-project results internally by way of a 2 page "fact sheet". The original intent was to have this document completed and circulated for the spring of 2013. Rolled-up numbers could be used to communicate post-program results externally. However, it was determined that the post-project data received for the March 31, 2013 period would not provide an accurate assessment of the long-term environmental benefits achieved via the Program, as project completion dates have varied and not all projects would have reported on actual benefits achieved over the course of a full calendar year.
CFS Staff assigned with closeout responsibilities for the PPGTP will integrate this project into a formal plan to utilize the data received, including the March 31, 2014 post-project reporting, to update this information in the form of a concise "Fact Sheet on PPGTP Performance" and release to NRCan and external stakeholders, as appropriate.
|ADM – CFS (September 2014)|
|4. NRCan should consider documenting the PPGTP life-cycle costs and other implications for CFS to inform future design and risk assessment for grants and contributions programs.||The CFS agrees with the recommendation and proposes to build on the existing PPGTP Lessons Learned Report to disseminate information on program close-out responsibilities and impacts. Information would include program administrative costs, staff time and salary, continued stakeholder interactions, and monitoring requirements based on CFS/PPGTP staff experience over the course of April 2012 to July 2013. Additionally, the analysis conducted would be available to help inform future program design within CFS and the department.||ADM-CFS (January 2014)|
|5. CFS should consider the opportunity to capture key socio-economic data and report back to Canadians for future industry-focused Programs with a national scope and large materiality.||CFS agrees with the recommendation and proposes to systematically consider the opportunity to collect on a mandatory basis key socio-economic data and report back to Canadians where applicable when designing future industry-focused funding programs. The level of burden will be commensurate with the materiality of the program.
Throughout the PPGTP, qualitative performance indicators on socio-economic parameters were a recommended (not mandatory) reporting component of the Performance Measurement Guide distributed to proponents.
In addition to reporting on the quantitative performance indicators, proponents are encouraged to report qualitatively (anecdotally) on any supplementary benefits of a PPGTP-funded project. This could include direct and indirect benefits of a project occurring onsite, as well as in the local community.
The program did not want to place undue burden on the recipient by making it mandatory to provide data beyond the mandate of the program, aligned under NRCan's PAA 2.1.5 Environmental Responsibility, although it was encouraged in the post-project environmental reports. The burden of reporting was heavily considered when drafting the Performance Measurement Guide and templates to ensure best alignment with the TBS Directive on Transfer Payments - Appendix B: Core Design Elements: 12. The administrative requirements on applicants and recipients and a strategy to ensure these are no more than is needed to meet the department's control, transparency and accountability requirements.
|ADM – CFS (ongoing)|
This report presents the findings, conclusions and recommendations for the evaluation of the Pulp and Paper Green Transformation Program (PPGTP).
The PPGTP was established by the Government of Canada (GoC) and implemented by the Canadian Forest Service (CFS) within Natural Resources Canada (NRCan). Its main goal was to improve the environmental performance of Canada’s pulp and paper industry by providing specific incentives to enhance industrial energy efficiency and increase renewable energy production. Within the NRCan Program Activity Architecture (PAA), this Program most recently represented sub-activity 2.1.5 under Environmental Responsibility.
The evaluation covers the period from fiscal year 2008-09 to 2011-12 and includes all the activities of the PPGTP, amounting to $975.5 million in expenditures. Given that the Program ended on March 31, 2012, this evaluation also aims to assess the factors underlying the success of the Program, as well as to identify best practices and lessons learned that can be of value for other initiatives within NRCan. The evaluation was conducted in accordance with the 2009 Treasury Board Policy on Evaluation and the Departmental Strategic Evaluation Plan (2012-13 to 2016-17).
The evaluation design and findings were also informed by feedback from an Evaluation Advisory Committee (EAC) composed of Program and industry representatives. Findings from multiple lines of evidence were integrated, resulting in preliminary findings which were then presented to the EAC for validation.
2.0 Program Background and Profile
In Canada, the forest sectorFootnote 5 contributes about 1.9% to the national Gross Domestic Product (GDP), with annual revenues worth $57.1 billion dollars. By value, Canada is the world’s largest exporter of softwood lumber, newsprint, and wood pulp, with the US being the largest buyer of Canadian forest products. Within the forest sector, the pulp and paper manufacturing sector contributed $8.5 billion to Canada’s GDP in 2011, and represented approximately 18% of national industrial energy use.Footnote 6,
During 2003–2013, the pulp and paper sector experienced a significant economic decline. Current prospects for industry recovery are mixed. There is continued demand for certain traditional products like bleached softwood pulp, while for others, such as newsprint and office paper, the outlook is less positive as electronic media force a decline in per capita consumption.Footnote 7
The pulp and paper sector is also among Canada’s largest producers of renewable energy, mainly through extensive production of wood waste that can be used as a fuel.Footnote 8 This waste includes black liquor, a by-product of chemical pulping processes that is routinely burned, both to recover the pulping chemicals for further use, and to generate renewable heat and power.Footnote 9
Capital investments to improve the industry’s environmental performance have been delayed by the economic circumstances facing the industry. Because the pulp and paper manufacturing process is energy intensive, improvements in energy efficiency and increases in the production of renewable energy have the potential to deliver significant environmental benefits, through reduced energy consumption and air emissions, and by improving water effluent treatment. It was in this context that the Government of Canada (GoC) created PPGTP to support capital investments in Canadian pulp and paper mills that would contribute to the environmental, and hence commercial sustainability of the industry.Footnote 10
2.2 Mandate and Stakeholders
Figure 1: PPGTP project distribution
Figure 1: PPGTP project distribution
An illustrative map of Canada displays the locations of PPGTP projects across Canada denoted by green dots. Projects were implemented in eight provinces: British Columbia, Alberta, Saskatchewan, Manitoba, Ontario, Québec, New Brunswick, and Nova Scotia. The map shows concentrations of projects in British Columbia and in Quebec/New Brunswick.
The objective of the Pulp and Paper Green Transformation Program (PPGTP) was to improve the environmental performance of Canada’s pulp and paper industry, in turn contributing to environmental, and hence commercial, sustainability.Footnote 11 More specifically, the Program assisted Canadian pulp and paper mills in making capital investments that would improve facility environmental performance, by providing a $0.16 per litre credit for black liquor produced at mills between January 1, 2009 and December 31, 2009, or to a cap of $1.0 billion in total credits issued, whichever came first.Footnote 12
The Program was open to any firm with mills in Canada running a chemical processing operation that produced black liquor.Footnote 13 Expected outcomes were improved energy efficiency, increased production of renewable energy, improved environmental performance and investments in innovation and technology to contribute to an environmentally and commercially sustainable pulp and paper industry.Footnote 14
Twenty-four companies, representing 38 pulp and paper mills across Canada qualified for credits under the Program (Figure 1). Based on approved submissions, firms were allocated a credit balance and had until March 31, 2012 (the official Program close date) to draw from their balance to finance approved capital projects.Footnote 15
The direct stakeholders in this Program were the Canadian, the pulp and paper industry, PPGTP funding recipients, and communities (including Aboriginal) in which projects were located.
2.3 Governance and Administration
Administratively, the PPGTP fell under the responsibility of the Assistant Deputy Minister (ADM) CFS, and was located in the Policy, Economics and Industry Branch of CFS at NRCan. The Program was led by a Director-General, while daily operations were overseen by a Program Director and a Program Manager. After the Program’s Terms and Conditions expired in March 2012, a small number of staff (2-4) within the branch were delegated with part-time and/or acting responsibilities for remaining close-out activities, such as the final report collection.
Specific authorities for the PPGTP include the Department of Natural Resources Act, S.C. 1994, c.41, section 5); and the Energy Efficiency Act, S.C., 1992, c.36, section 21 (d). The applicable legislation for the release of payments is the federal Financial Administration Act (FAA), and payments were to be made in accordance with the Treasury Board Policy on Transfer Payments.
2.4 Activities, Delivery and Organization
Under the PPGTP, black liquor was defined as including all liquid by-products of chemical pulping processes having positive caloric values and burned in recovery boilers for the purpose of energy production.Footnote 16 Firms had until March 31, 2012 to use allocated black liquor credits to finance approved capital projects. Credits earned at one pulp and paper facility could be spent upon eligible projects at another facility owned by the same company.Footnote 17 Credits were awarded on a “first-produced, first-credited” basis.Footnote 18
Funding recipients were required to provide regular progress and financial reports, as well as more detailed annual reports.Footnote 19 Recipients under the PPGTP were also required to identify all related sources of funding and to submit an Environmental Impact Statement (EIS) as part of the environmental assessment process to provide information on the project’s overall environmental impact.
All projects were screened by NRCan, in consultation with external experts such as those at Environment Canada (EC), to ensure that they met the Program requirements of improved environmental performance (including enhanced energy efficiency or increased renewable energy production). Reviews were conducted at various stages including for proposal submission, technical feasibility, project risk, consistency with the PPGTP’s terms and conditions and Canada’s international trade commitments, environmental assessment and Aboriginal consultation.Footnote 20 Finally, proponents were also required to submit progress reports on a quarterly basis, produce a comprehensive final report, and provide Environmental Benefits reporting pre-project, upon the conclusion of the project, and at one and two years after the project end date. Close-out activities after the Program end date included collection and review of these reports, remaining technical/financial audits and/or amendment adjustments and activities in preparation for this evaluation. Figure 2 illustrates key points in the Program’s timeline.
Figure 2 Key Program dates
Figure 2 Key Program dates
June 2009: Program Announced
August 2009: Applicant’s Guide released
September 2009: Deadline to Register with Program
September / October 2009: Calculation of Credits
October 2009: Credits awarded
October 2009 to september 2011:
Proposals were submitted and reviewed by PPGTP.
Project implementation begins.
September, 2011: proposal submission ends.
March 31, 2012: Program end date
2012-13: Close-out phase
Source: Adapted from Pulp and Paper Green Transformation Program Report on Results.
2.5 Funding and Resources
Resources for the PPGTP were allocated over three fiscal years, from 2009-10 to 2011-12 inclusive. The majority was consumed during the last two years, when the bulk of the project assessment and implementation work was carried out. Actual Program spending was 97.6% of the authorized spending authority of $1 billion. The planned and actual resources for PPGTP are presented in Table 1. Note the Program’s budget did not extend into 2012-13. Resources for Program activities in 2013-14 stem from other departmental funds.
|2009-10 ($ Millions)||2010-11 ($ Millions)||2011-12 ($ Millions)||2009-12 ($ Millions)|
Note: * does not include Accommodation or Employee Benefit Plan
** numbers are rounded to the nearest million; additions may not be exact due to rounding
Source: Provided by NRCan Finance and Administration, Policy Economics and Industry Branch
Particular care was taken in the approach to staffing the PPGTP. Hiring was based on acquiring specific skills for identified tasks, to ensure a full range of expertise was available to the Program within the condensed timeline. The Program generally averaged 16 full-time equivalents (FTE) per fiscal year. Positions included a mixture of tenures (e.g., contractual, term, executive exchange). Service contracts and Memoranda of Understanding were used to access external skills and expertise in the areas of environmental assessment, technical audits, and other services (e.g., translation).
The Program was headquartered at NRCan in Ottawa, Ontario. While there were no regional offices, PPGTP staff travelled to mills to conduct site visits on an as-needed basis. In addition, two regional workshops were held for proponents (in Montreal and Vancouver) during the project application stage.
2.6 PPGTP Logic Model
The Program theory and logic model for the PPGTP (as presented in Figure 3 below) was refined from the logic model originally presented in the Program’s Results-Based Management and Accountability Framework (RMAF).Footnote 21 The refinements were made to illustrate the inputs and outcomes of the Program more clearly, and to help identify indicators for the evaluation. These revisions were informed by the PPGTP Audit,Footnote 22 PPGTP Report on Results,Footnote 23 and the EAC members. The Logic Model is presented in categories numbered from 0 to 6 as follows:
0. Influences/External Factors: A number of external factors identified as relevant to the PPGTP design are described in boxes 0.1 to 0.4. Notably these include an increased focus on environmentally sustainable practices and competitive pressures such as the 2008 economic downturn. These were also reflected in the policy context at the time (i.e., the Clean Air Agenda and the Economic Action Plan).
1. Resources: Boxes 1.1 to 1.7 detail the inputs from both the department and the industry stakeholders. These include personnel, operations and maintenance (O&M) and the funding contributions provided for the projects (boxes 1.1 to 1.3); the capital investments from the proponents themselves and other sources (1.4); and also various management processes and technical or administrative support (see boxes 1.5 to 1.7).
2. Activities: All Program activities, including establishment of guidelines, determination of company eligibility, calculation of black liquor credits, review of proposals, and communication plans are outlined in boxes 2.1 to 2.8.
3. Outputs: Conceptually linked to Program activities, are the outputs described in boxes 3.1 to 3.6, such as guides, signed Contribution Agreements, project performance reports and all related correspondence or documentation.
4. Immediate Outcomes: Immediate outcomes are expected to be produced in the short-term. These included increased capital investments at eligible mills (box 4.1), as well as stakeholder awareness and engagement leading to participation in, and support for, the Program by proponents (box 4.5). The approved projects were to be completed by March 31, 2012, and were designed to include the implementation of: i) improved controls for emissions, waste and pollution, and water usage (box 4.2), ii) processes to improve energy efficiency (box 4.3), and/or iii) processes to increase the production of renewable energy (box 4.4).
5. Intermediate Outcomes: Boxes 5.1 to 5.5 present the Program’s intended outcomes once projects have been implemented. These included reduced air emissions (box 5.1), reduced impacts on water and land (box 5.2), cost reductions as a result of the environmental investments (box 5.3), new revenue streams (box 5.2), as well as various socio-economic benefits (e.g., jobs, community, aesthetic; box 5.5)
6. Long-Term Outcomes: The ultimate outcome of the PPGTP is presented in box 6.1. Namely, investments in innovation and technology were expected to have a transformative effect by contributing to an environmentally sustainable pulp and paper industry in Canada in the long-term, and thereby improving the industry’s commercial sustainability
Figure 3 PPGTP logic model
Figure 3 PPGTP logic model
The Logic model presents the program theory of the PPGTP in graphic form. It illustrates the linkages among a program’s elements: activities, outputs (tangible deliverables), immediate outcomes, intermediate outcomes, and long-term or final outcomes, which are represented by rows. This logic model also contains a row for resources, and has a row depicting the major internal and external factors that influenced the program’s development and design.
Each row is numbered. This logic model does not have lines or arrows connecting the elements of the program (activities, etc.) because there is no one-to-one correspondence between the elements. Several activities, for example, could contribute to one output. These linkages are indicated by feedback loops.
Each row is numbered from 0 to 6 and its contents are numbered and titled as follows:
0. Influences / External Factors
Influences / External Factors element identifies the major internal and external factors that influenced the PPGTP’s development and design.
There are four main Influences / External Factors shown in layer 0. They are:
0.1 Increased competitive pressures within the global pulp and paper sector
0.2 Economic downturn. Canadian pulp and paper producers were thus faced with capital constraints that limited their capacity to invest in their facilities.
0.3 Increased focus on environmentally sustainable practices. E.g. carbon markets.
0.4 Canadian policy context under which the program was created (e.g., Clean Air Agenda and Economic Action Plan).
The resources layer identifies seven areas of input into the program. These include not only those provided by NRCan, such as personnel, operating & maintenance (O&M) and the contributions provided for the projects (boxes 1.1 to 1.3), but also capital investments provided by the proponents themselves and other sources (1.4). Similarly, other inputs included various forms of governance/management processes and technical and administrative support to design and implement the program within NRCan (boxes 1.5 and 1.6), as well as the project oversight and management processes provided by the project proponents (i.e., industry, box 1.7).
The seven areas of input are:
1.2 Operating & maintenance (O&M)
1.3 Grants and Contributions (G&C)
1.4 Industry investments, resources and facilities provided by the proponents themselves and other sources.
1.5 Program governance/management processes
1.6 Technical and administrative support (including independent committees)
1.7 Industry project oversight and management processes provided by the project proponents
The Activities element of the PPGTP logic model is comprised of six major areas.
2.1 Establish program guidelines and processes
2.2 Determine eligibility, calculation and administration of black liquor credits.
2.3 Review of project proposals (technical reviews, environmental assessments and Aboriginal consultations.)
2.4 Prepare contribution agreement (expert reviews, CA development, negotiation and finalization)
2.5 Project monitoring (baseline / post) and audits
2.6 Stakeholder outreach and communications
In logic models, outputs are tangible results of activities. There are six outputs that roughly correspond to the activities with the same number.
3.1 procedures, guides and committees
3.2 database of eligible companies and their black liquor credits
3.3 Environmental assessments and project reviews
3.4 contribution agreements and transfer payments
3.5 performance data, project/program audits, as well as various reports
3.6 Announcement, workshops, correspondence and documentation
4. Immediate Outcomes
The immediate outcomes portion of the logic model shows the short term results of the program activities and outputs. What ‘short’ means varies from program to program and is a relative term (i.e., short term occurs before mid-term which precedes long term). Usually short term is three to five years, long term is ten and more years, and mid-term ranges from five to ten years.
There are five immediate outcomes noted in the logic model for PPGTP. They are:
4.1 Increased and targeted capital investments
4.2 Implementation of control for emission, waste and pollution and water usage
4.3 Implementation of processes to improve energy efficiency
4.4 Implementation of processes to increase production of renewable energy
4.5 Stakeholder awareness and engagement
5. Intermediate Outcomes
In the intermediate timeframe – once the capital investments have been made and the projects implemented – the PPGTP-supported improvements were expected to directly and indirectly contribute to outcomes in five areas. They are:
5.1 Reduced air emissions
5.2 Reduced impacts on water and land
5.3 Cost reductions flowing from environmental investments (fossil fuel and electricity savings, other savings)
5.4 New revenue streams flowing from environmental investments (electricity sales, carbon markets, new products)
5.5 Socio-economic benefits (e.g., jobs, community, aesthetic)
Note that the assessment of intermediate outcomes should take into account that not all projects are expected to lead to all of these outcomes.
6. Long-Term Outcomes
The expected long-term outcome of the PPGTP is that the investments in innovation and technology have a transformative effect. It is stated as:
6.1 Environmentally sustainable pulp and paper industry in Canada, resulting in a more commercially sustainable industry
A feedback loop connects the Influences and External Factors (layer 0) to the Resources (layer 1) used by the PPGTP. A second feedback loop exists between the Outputs (layer 3) and the Immediate Outcomes (layer 4). A larger feedback loops joins these two feedback loop to show the iterative flow of the program across the various layers.
Source: Adapted from the logic model in the PPGTP Horizontal Results-Based Management and Accountability Framework. Revisions were informed by the PPGTP Audit, Report on Results, as well as by the EAC members.
3.0 Evaluation Objective, Scope, and Methodology
3.1 Objectives and Scope
The purpose of the evaluation was to assess issues relating to the relevance and performance of the PPGTP and provide recommendations as necessary. All steps within this evaluation were designed with the intent of producing a final evaluation report that met the Treasury Board Policy on Evaluation and its supporting Directive and Standards.Footnote 24
The evaluation covered the period from 2008-09 to 2011-12 and included all the activities of the PPGTP, which represent approximately $1 billion in expenditures. Given that the PPGTP ended on March 31, 2012, this evaluation also aimed to assess the factors underlying the success of the Program, as well as identify best practices and lessons learned that could be of value for other initiatives within NRCan. The intent was that the evaluation results could effectively inform future decision-making and Program development in the context of similar challenges or initiatives.
This evaluation addresses the five core evaluation issues defined by the Treasury Board Secretariat in the Directive on the Evaluation Function, effective April 2009:
Relevance of the PPGTP:
- Continued need for Program,
- Alignment with government priorities , and
- Alignment with federal roles and responsibilities.
Performance (effectiveness, efficiency and economy) of the PPGTP:
- Achievement of expected outcomes, and
- Demonstration of efficiency and economy.
The evaluation approach for this project was developed in consultation with the Evaluation Advisory Committee (EAC.) It was designed to draw on various sources of data to ensure that the combined lines of evidence resulted in an in-depth and comprehensive analysis. A data collection matrix was used to cross-link evaluation questions with associated performance indicators, data collection methods and data sources, allowing data to be triangulated and compared for each question during the analysis. This practice helps ensure the production of a sound, defensible and useful evaluation report.
The evaluation approach included three main phases:
- The design phase, which included the development of the methodology and tools. These were validated in a workshop with the EAC.
- The evaluation fieldwork and technical analysis phase, which included all data collection activities and preliminary data analysis by method.
- The integration and reporting phase, which encompassed the integration and analysis of data across methods that led to the production of this evaluation report.
Four methods were used to collect and analyze evidence. Table 2 presents additional details on each data collection method:
- document file and data review,
- 30 interviews,
- online survey of funded project participants, and
- three case studies on Program planning, management and delivery.
|Data Collection Method||Details|
|1. Document, file and data review||A review of approximately 500 documents, secondary literature, files and Program data included:
|2. Stakeholder interviews (internal and external)||Total number of interviews................................................................. 30
|3. Web survey||Online survey of funded project participants
|4. Management and delivery case studies||Case studies.......................................................................................... 3
Note: † Valid response rate = Number of completed surveys, divided by the valid sample, which excludes unreachable potential respondents; ‡ Calculated for a response distribution of 50% (i.e., 50% yes/50% no); 95% confidence level (19 times out of 20)
3.3 Evaluation Challenges, Limitations and Mitigation Strategies
The challenges and data limitations encountered during this evaluation, and the mitigation strategies adopted to counter them, are discussed below. Generally, challenges were anticipated early in the process and associated mitigation strategies were proactively built into the evaluation design. Essentially, the evaluation was designed to use multiple lines of evidence to answer each evaluation question. Findings were then validated over different sources. Specific data limitations were identified, including some that were linked with the scope of the evaluation project. The amount of program data available, the high level of participation from identified respondents, and the triangulation of findings, all contributed to minimizing the effects of these limitations on the evaluation results.
Evaluation design: This evaluation faced a particular challenge in that it would not inform Program direction or renewal since the PPGTP had ended. Therefore, the evaluation sought to obtain information about elements of Program design that a) facilitated implementation of a large-scale initiative responding to an acute need, b) was delivered in a short period of time and c) would be useful if a comparable initiative was undertaken in the future. Moreover, while the evaluation was designed to assess both relevance and performance, more emphasis was placed on collecting performance information, particularly on elements of Program design and planning that affected the outcomes. This ensured that recommendations were valuable for CFS and NRCan going forward.
Data collection: Typical evaluation challenges were apparent throughout the data collection stage. These included the level of effort required to gather and address input on data collection tools from multiple EAC members, scheduling of interviews over the winter holiday period, prioritizing the wealth of Program data provided for the evaluation while data was being updated in real-time, as well as seeking out appropriate initiatives for case study and securing their participation. Regular communication between the evaluation team, SED and EAC members as necessary allowed any issues to be addressed in a timely manner.
For the case studies in particular, it is important to note that since only two alternative initiatives were examined, this limited the ability to extract findings across a wider range of contexts. However, even these two provide a valuable source of evidence to examine the management and delivery of large-scale, industry-focused Programs. As the PPGTP was unique in its political context, its large materiality, its national scope and its targeted sector focus, the case study method did not seek to make any direct comparisons with other Programs. The case studies thus do not serve as a benchmarking exercise. Findings that discuss the PPGTP and the two alternative initiatives should be interpreted with a view to contrast the findings on the management and delivery of the PPGTP and to identify best practices and lessons learned that could be of value for other initiatives within NRCan.
Attribution: It is challenging to attribute Canada-wide environmental outcomes (e.g., GHG reductions) to a single project or program as there may be complementary projects or processes affecting the outcomes. Mitigation in this respect relies on the Program’s theory of change to focus on the contribution of the Program/projects, rather than attributing Canada-wide environmental outcomes directly to the Program.
3.3.2 Data limitations
Survey: Given the small survey population and the diversity of survey questions, there was a possibility that targeted respondents were not able to respond accurately to some survey questions. In fact, questions asked ranged from administrative and financial information to project results and other outcome data. Consequently, a few industry representatives signalled to the evaluation team that more than one individual within their organization answered the survey to account for their respective area of expertise. Nevertheless cross-analyses in the survey (by project type and region) were closely aligned to performance outcomes noted in other lines of evidence and thereby lending a measure of validation to survey results.
Potential bias: In all consultation-based data collection in which stakeholders’ perspectives and experiences are an important data source, there is a potential for data bias. In the case of the PPGTP evaluation, the Program had ended and a Report on Results had been already distributed when the evaluation data was being collected. The potential existed that interviewees may have simply reflected the common Program storyline disseminated since Program closure or sought to focus on positive Program features and outcomes, requiring mitigation. Similarly, the perspectives of industry representatives interviewed and surveyed might have been influenced by the important size of funds provided to the industry and motivated by potential future funding opportunities. By using multiple lines of evidence, the qualitative information and views were balanced by quantitative data on Program outcomes, and the potential for biased conclusions was minimized.
Case studies: Given the evaluation scope, the limited number of management and delivery case studies (i.e., the number of different initiatives reviewed) that could be conducted limited the ability of this evaluation to fully contrast the PPGTP findings and reflect the wide range and complexity of practices within the federal community. In addition, direct consultations with beneficiaries and external stakeholders would have provided a more balanced perspective on the management and delivery models used by the initiatives (i.e., Strategic Aerospace and Defence Initiative [SADI] and Farm Income Payment Program [FIPP]). However, as noted above, within the scope of the project, the case studies confirmed PPGTP lessons learned and documented by the CFS, and identified additional practices of interest for the federal community.
4.0 Evaluation Findings
4.1.1 Continued Need for the Program (Issue 1)
Question: Did the PPGTP address demonstrable needs?
Yes. The PPGTP provided access to capital financing directed towards projects that would improve environmental performance of the industry and thereby strengthen economic viability at a time when the industry was severely challenged to remain globally competitive.
All lines of evidence confirmed that the years leading up to Program implementation were some of the most difficult in recent history for the Canadian pulp and paper industry. Net revenues in the Canadian Paper Manufacturing subsector decreased between 1998 and 2007Footnote 25 and subsequently, Canadian companies were especially hard hit by the global economic recession in 2008, as marked by a 355% increase in net losses.Footnote 26 Much of the sector’s loss was attributed to a decline in newsprint demand, the entrance of new low-cost competitors such as Brazil and Indonesia, as well as volatility in currencies and input costs.Footnote 27 Tight credit markets meant that companies had limited access to the financing required for critical upgrades to aging equipment and to address expanding environmental concerns.
These views were reflected by industry stakeholders. For example, 75% of evaluation survey respondents agreed that there was an important need to improve the environmental sustainability of their organization at the time of PPGTP initiation. Moreover, almost all survey respondents agreed that there was an important need to improve the economic competitiveness of their organization’s facilities via environmentally-focused capital investments at the time (Figure 4).
Secondary to these contextual factors, a US tax credit led to a market distortion in the North American pulp and paper industry. As described in the Program’s Results-Based Management and Accountability Framework (RMAF),
In 2008, US pulp producers discovered a loophole in the Alternative Fuels Mixture Tax Credit (AFMTC) that allowed them to apply for a USD $0.50 per gallon refundable tax credit for the production of a mixture of diesel and black liquor. Payments under this Program were estimated at USD $7-9 billion to December 31, 2009, at which time the Program expired.
An internal PPGTP planning document indicated that in the long-term the US tax credit would, “recapitalize US producers enough for them to invest in machinery upgrades that could permanently alter the competitive balance in North America.”Footnote 28
Figure 4 Perception of industry representatives on the extent to which the PPGTP addressed the needs of their organization
Figure 4 Perception of industry representatives on the extent to which the PPGTP addressed the needs of their organization
Figure 4 shows the percentage of survey respondents (N=55) who agreed (combined agree and strongly agree) with four listed questions.
The four questions are:
- PPGTP as it was implemented met the needs of my organization: 98%
- There was an important need to improve the economic competitiveness of my organization’s facilities via environmentally-focused capital investments: 98%
- There was an important need to improve the environmental sustainability of my organization. 75%
- PPGT objectives listed above were well aligned with the needs of my organization. 100%
Source: Survey data with PPGTP industry project representatives (n=55 respondents)
Considering these factors, the PPGTP clearly addressed the most critical needs of the pulp and paper industry by providing rapid access to capital financing geared towards environmentally-focused investments. Environmental sustainability would therefore contribute to Canadian companies remaining globally competitive at a time when the industry was severely challenged.
Now that the Program has ended, the industry is looking to address future needs. For example, the Forest Products Association of Canada (FPAC) has said its members are moving towards engaging a renewed workforce, a 35% improvement to the sector’s environmental footprint, and an additional $20 billion in economic activity from new innovations and growing markets.Footnote 29 Interview and survey evidence supported this view, as stakeholders indicated that they aimed to fill this need for transformation with their own in-house resources (e.g., by selling electricity back to the grid, by continuing to work on internal efficiencies) and by seeking out continued funding opportunities at the provincial and federal government level.
4.1.2 Alignment with Government Priorities (Issue 2)
Question: Was the PPGTP consistent with government priorities and NRCan strategic objectives?
Yes, the PPGTP’s expected outcomes were clearly aligned with federal priorities and NRCan strategic objectives throughout the evaluation period.
Federal priorities over the evaluation period (e.g., Speeches from the Throne 2008 to 2011) consistently displayed a commitment to developing the forest industry.Footnote 30 This commitment was reflected in the main goal of the PPGTP to support green capital investments in Canadian pulp and paper mills that contributed to the environmental, and hence commercial, sustainability of the industry. Furthermore, the government’s Clean Air Agenda (CAA), ongoing since 2008, is a national approach to decrease air pollutants and greenhouse gases, and supports investments to reduce industrial emissions.Footnote 31 In this context, the Program’s main expected outcomes (e.g., improved energy efficiency, renewable energy production, and environmental performance) clearly support the CAA.
The federal government’s Economic Action Plan (EAP) has also supported investments towards innovation and diversification in the forestry sector. This included over $200 million (from Budget 2009 and 2011) for forest innovation and market development.Footnote 32 While the PPGTP was not formally part of the EAP, it was expected to encourage investments in forest innovation and technology. Further, the PPGTP was implemented in two years, which aligned with EAP timelines. Thus it is clear that the Program’s design was influenced by, and closely reflected, the EAP’s general objectives.
Regarding departmental objectives, internal interviewees agreed that the PPGTP was aligned with NRCan goals as outlined in the Program Activity Architecture (PAA) at the time, under Strategic Objective #2, Environmental Responsibility. This outcome states that Canada is a world-leader on environmental responsibility in the development and use of natural resources. Similarly, the Program’s RMAF notes that there was alignment with the broader NRCan mandate to, “enhance the responsible development and use of Canada’s natural resources and the competitiveness of Canada’s natural resource products.”Footnote 33 Finally, the Program’s goals also reflect the CFS mission to, “promote the sustainability of forests and a competitive forest sector to improve the quality of life of Canadians.”Footnote 34
4.1.3 Alignment with Federal Roles and Responsibilities (Issue 3)
Yes. For issues concerning international trade, the GoC has a clear constitutional responsibility and therefore a legitimate role to protect Canadian market interests. Contribution Agreements were found to be an appropriate mechanism to hold industrial recipients accountable to environmental outcomes. Further, in the context of serious market distortions at a time of reduced access to credit globally, the Program was necessary to induce improved environmental performance.
It is apparent that the federal government took legitimate and appropriate action both domestically and internationally to implement a targeted pulp and paper Program that promoted the sustainability of the forest sector.
At the time of Program design, industry stakeholders – represented by FPAC – were calling for federal action to support green transformation, ensure access to credit and to counter US subsidies.Footnote 35 There was also ample documentary evidence Footnote 36 (e.g., Question Period notes, Program committee notes) showing that the GoC was aware of the potential for a competitive imbalance in the pulp and paper sector due to the US Alternative Fuels Mixture Tax Credit (AFMTC). Since the federal government had jurisdiction over domestic and international market access issues according to the Department of Natural Resources Act,Footnote 37 it voiced its opposition to the tax credit publicly and in diplomatic circles.
There was recognition that any Canadian action had to comply with the terms of the Softwood Lumber Agreement (SLA). As such, the Department of Foreign Affairs and International Trade (DFAIT) advised NRCan to focus on environmental aspects as a key element of any planned Program or support effort.Footnote 38 NRCan confirmed that it would work with industry to consider options, and the PPGTP was subsequently announced in June 2009.Footnote 39
Interview and documentary evidence also indicated that the use of Contribution Agreements, which contractually obliged PPGTP recipients to environmental objectives, was superior to the US tax credit model,Footnote 40 and appropriate to the role of the GoC, especially when considering the terms of the Softwood Lumber Agreement. Each CA limited Canadian companies to using the funds only for agreed purposes, whereas the US model imposed no conditions on funding.Footnote 41 The PPGTP was described as “smart government policy” because it was designed to provide a larger environmental benefit. Footnote 42
Finally, the interview, documentary and case study evidence confirmed that while several Programs for the forest industry were operationalized at the same time as the PPGTP, none specifically targeted pulp producers, had the same environmental goals, nor had the same materiality as the PPGTP.Footnote 43As well, the Stimulus Phase of the Canada’s EAP was launched in January 2009 and directed large investments towards various other industrial sectors including automotive, agriculture, mineral exploration, and infrastructure.Footnote 44 In this respect, some external stakeholders indicated that the federal government’s response towards pulp and paper was “almost too late” considering the strain on the industry and the stimulus funding that went out to some other sectors earlier in the same year. Based on the sense of urgency around Program implementation, external interviewees suggested that the federal government should aim to play a more analytical, proactive role to continuously support a sustainable industry, rather that reacting later when access to capital had become an urgent issue.
4.2.1 Achievement of Expected Outcomes (Issue 4)
Question: To what extent have the PPGTP’s intended outcomes been achieved?
The PPGTP’s intended outcomes have been achieved to a great extent and all eligible pulp and paper mills were engaged via targeted outreach efforts. Program environmental targets were met or exceeded for energy savings and for the production of renewable electricity, while other improvements (e.g., reduced emissions, water use, chemical waste) were also documented. While various internal facilitators contributed to this achievement, the Program’s short timeframe was a challenge for project implementation and Program delivery. Finally, the PPGTP appeared to have contributed to the longer-term environmental and commercial sustainability of the pulp and paper industry.
The PPGTP’s intended outcomes have been achieved to a great extent. Supporting evidence is presented under individual subheadings below.
Industry participation: Overall, 24 companies were allocated credits under the PPGTP based on the production of black liquor at 38 mills across Canada.Footnote 45 This represented the entire eligible pool of mills (i.e., mills in Canada producing black liquor between January 1 and December 31, 2009).Footnote 46 Of 102 proposals received, 98 projects were funded (96%). The Program also processed 71 amendment requests for items such as a change in project timelines, recipient/company name or project scope.Footnote 47 The distribution of projects generally followed the distribution of chemical mills across Canada with the majority of projects located in British Columbia, Quebec, and New Brunswick. The total project values were highest for British Columbia, Alberta and Quebec (see Figure 5). The average PPGTP contribution per project was $9.7 million.
Interview and survey data indicated that a majority of projects were already identified by companies before the PPGTP announcement; in other words, projects were in a preliminary or planning stage, but implementation could only be continued if financial resources were available.Footnote 48 This confirmed a recognized need for these projects by industry and subsequently expedited project selection and implementation once the Program was formally announced.
Figure 5 Distribution and value of PPGTP-funded projects
Figure 5 Distribution and value of PPGTP-funded projects
Figure 5 is a bar graph showing the distribution and approximate value of PPGTP – funded projects. The provinces are listed left to right in order of total value of the projects; the amounts are illustrated, not stated, but the range is from zero dollars to four-hundred and fifty million dollars.
- A total of 28 projects were carried out in British Columbia an approximate value of just over four-hundred million dollars.
- A total of 7 projects were carried out in Alberta at almost two-hundred million dollars.
- A total of 22 projects were carried out in Quebec at an approximate value of one-hundred and thirty-five million dollars.
- A total of 18 projects were carried out in New Brunswick at an approximate value of ninety-five million dollars.
- A total of 11 projects were carried out in Ontario at an approximate value of seventy-five million dollars.
- A total of 3 projects were carried out in Nova Scotia at an approximate value of thirty million dollars.
- A total of 8 projects were carried out in Manitoba at an approximate value of fifteen million dollars.
- One project was carried out in Saskatchewan at an approximate value of several million dollars.
Source: Compiled by Science-Metrix using Program data: PPGTP Final Project Values and Contacts (October 2012)
Targeted outreach: The Program had a large and well-organized record of outreach activities spanning its entire lifecycle. For example, there was clear documentary evidence of presentations made to trade and research associations to increase industry awareness (e.g., FPAC, Footnote 49 PAPTAC, Footnote 50 Pacwest conferencesFootnote 51), to other federal and provincial government departments to coordinate advice (e.g., Environment Canada, Footnote 52 provincial environment ministriesFootnote 53), to internal stakeholders so that administration processes supported Program needsFootnote 54 (e.g., NRCan Transfer Payment Review CommitteeFootnote 55), as well as internationally to promote environmental sustainability (e.g., FAO European Forestry Commission). Footnote 56 The Program held several workshops and webinarsFootnote 57 in 2009 to guide proponents through the various stages of project application and implementation (e.g., explaining timelinesFootnote 58, project eligibility, the Environmental Assessment (EA) process,Footnote 59 and performance measurement reporting).Footnote 60 These sessions were complemented by information on the CFS website and via dissemination of clear guidelines such as the PPGTP Applicant’s Guide and Performance Measurement Guide.Footnote 61In August 2010, the Program sent a series of letters to companies with an approved black liquor credit allocation but that had not yet forwarded a formal project proposal. Companies were encouraged to do so and reminded of the September 30 deadline.Footnote 62 Finally, outreach to the general public in the form of official press releases and/or media events occurred for almost all of the 98 projects.Footnote 63
The most significant results were realized for energy savings (i.e., nearly doubling the PPGTP initial target) and production of renewable electrical energy (i.e., exceeding the PPGTP initial target by 36%.) As well, energy savings resulting from PPGTP projects were expected to reduce mills’ GHG emissions by about 543,000 tonnes per year.Footnote 64 In addition to these parameters, data were collected on several other environmental goals before, during, and after the project.Footnote 65 Quantitative improvementsFootnote 66 for these goals were also mentioned by interviewees as well as documented in project final reports and in the Program’s formal Report on Results.Footnote 67
|Outcomes||Target (1)||Achieved (2)||% of target achieved|
|Energy Savings (GJ/year)||4,230,000||8,415,141||199%|
|Renewable energy(3) (MWh/year)||2,100,000||2,863,708||136%|
Source: (1) NRCan Program Activity Architecture/Performance Measurement grid 2011-12 PAA-PMF development template PPGTP (Sub activity 2.1.6); (2) Internal Program data (rolled up PPGTP benefits report, October 2012 and Excel Spreadsheet File entitled Figures Feb 1_compressed); (3) Includes renewable electricity (achieved 1.66 MWh/yr) and renewable thermal energy (achieved 1.23 MWh/yr)
Facilitators and barriers: Across all lines of evidence, several facilitators enabling achievement of Program outcomes were identified. The most important were as follows.
- Clear political will and senior-level support – Internal and external interviewees noted that the support of senior-level stakeholders from both government and industry was a key driver for quick implementation of a Program targeted specifically to the pulp sector. Documented examples of PPGTP-related correspondence to the Assistant Deputy Minister (ADM), Deputy Minister (DM) and Minister indicated clear engagement at these levels.
- Diverse in-house expertise – It was apparent that Program management recognized the importance and necessity of having a diverse range of experts involved to facilitate smooth and competent Program delivery. By using secondments, consultants and taking advantage of the Interchange Canada Program, the PPGTP was staffed with people knowledgeable in pulp and paper engineering, Aboriginal consultation, environmental assessment, grants and contributions, as well as policy analysis.
- An eligibility-based, 100% funding model with flexibility on location of investment – As the black liquor calculation was established before Program launch, companies knew exactly how much funding they were eligible to receive, and were not required to “top up” with any leveraged funds. Project planning was therefore undertaken with financial resources essentially secured, (as long as the projects met the Program’s criteria for improvements to environmental performance.) In addition, the flexibility to spend earned credits at any facility in Canada owned by the same company allowed investments that supported the strategic business direction of the company. Thus, the PPGTP funding attracted the interest of the industry, removed the need to consume resources negotiating amount of funding, and maximized the opportunity to contribute to the environmental and commercial sustainability of individual companies.
The most significant challenge identified by all lines of evidence was the short timeframe for project application and implementation. Interviewees indicated two main issues. First the increased demand in a limited time created bottlenecks in terms of availability of equipment/suppliers/tradesmen/ in construction seasons. Second, it influenced companies to “pick the low hanging fruit”, (i.e. to implement the simplest projects rather than more riskier projects with potential for higher pay-off.) Some external interviewees also suggested that a longer timeframe might have encouraged more transformative projects, even though they understood it was clear from the outset that no project extensions would be given.
On the point of Program duration, 70% of survey respondents who answered an open-ended question noted that the limited time to implement projects resulted in increased difficulties, particularly for projects that were approved in the final years of the Program. This sometimes resulted in higher implementation risks or reduced project quality. For example, additional costs were added, fewer preliminary tests were conducted, and/or the projects were sometimes implemented in unfavourable seasonal conditions.
Documentary evidence indicated that the PPGTP management was aware of these concerns from industry stakeholders and a rationale for a Program extension (e.g., an extension would allow for more transformative investments, more leveraging and could address supplier delays) was developed. However, the Program was not extended.
A secondary challenge to project implementation cited by interviewees, survey respondents and noted in the documentation was the length of the project review process (including the environmental assessments). The fact that projects had to cycle through EAs (with which most proponents were inexperienced) and that Contribution Agreements had to pass a series of reviews (e.g., legal, TPRC even for minor amendments) was perceived as burdensome. However, these processes were also generally viewed as necessary and not insurmountable or significantly hindering project success. Mitigation measures (such as the aforementioned workshops explaining the EA process) helped reduce the burden for industry stakeholders.
Longer-term outcomes: To estimate the impact of the PPGTP on the longer-term commercial sustainability of the pulp and paper sector, an analysis of factors contributing to overall return on investment (ROI) was conducted by the firm Science-Metrix. The analysis found that the Program clearly made a sizeable contribution to the commercial sustainability of the participating companies as the dollars generated or saved by the mills constituted a net income after four years. However there was considerable variability across PPGTP projects, as ROI ranged from -1% to 1339% of the original value of the investment on a yearly basis. For the industry as a whole, the one-time investment of $1 billion would be fully recovered four years later with a total (for all projects combined) yearly ROI of about $320 million (or 26% of the initial investment).Footnote 68
With respect to the environmental sustainability, the evaluation team conducted an exploratory factor analysisFootnote 69 plotting various project characteristics (e.g., project type, geographical location of the mill, mill size based on employee numbers, level of investment) against reported environmental benefits. Two main results emerged. It should be stressed that there was a lack of strong association patterns in the dataset; thus these results must be interpreted as indicative.Footnote 70
- Projects generating what are known as “oil savings” usually reduced their GHG emissions, as well as produced renewable thermal energy. The projects which performed best with respect to this dimension are those on boilers in the mid-size investment range (between $13 and $37 million), usually in Atlantic Canada. As these boiler projects were consistent with GHG reductions, it was likely that improvements to this type of technology were amenable to achieving such outcomes in the future.Footnote 71 Additionally, for the industry as a whole, it was found that PPGTP projects collectively contributed to reducing the consumption of heavy oil by 135 million litres a year. This accounted for as much as 45% of the total Canadian pulp and paper industry consumption (which is about 300 million litres a year).Footnote 72 Since there was a moderate association between this outcome and the reduction of GHG emissions, it could safely be inferred that the Program appeared to have had an important impact on the environmental sustainability of the pulp and paper industry.
- Projects with larger investments (i.e., $37 to $108 million) more often produced renewable electrical energy and had higher ROI ($/year). The projects which performed best within this dimension were those on turbines and turbine-boilers (usually located in British Columbia and the Prairies). Aside from the fact that larger investments were somewhat expected to generate larger ROIs ($/year) due to their scale, this might also be explained by the fact that production of renewable electrical energy, (as compared to thermal energy) was a more effective way to generate revenue by exporting this by-product to the grid. Even if these types of projects did not necessarily imply greater individual ROI in percentage of the initial investment, the return they generated in absolute terms was potentially higher. This should benefit the industry for years to come once the initial investment is fully recovered.
Note: Considering the rather small sample sizes and the unequal representation of project types, the above statements should be considered with care. Furthermore, it should be noted that these findings rely on moderate to weak associations between variables. As such, they should be viewed as hypotheses rather than as certainties.
Question: Were there any unintended (positive or negative) outcomes?
Yes, two main positive unintended outcomes were observed: internal capacity building at CFS and relationship building between CFS and the pulp and paper industry stakeholders.
Two main positive unintended outcomes were observed across lines of evidence – capacity building and relationship-building. These are discussed in further detail below. Additionally, Program documents and survey evidence showed signs of some ancillary benefits such as job creation and increased competitiveness (see Figure 6).
Figure 6 Perception of industry representatives on indirect or unintended outcomes observed as a result of the PPGTP project(s)
Figure 6 Perception of industry representatives on indirect or unintended outcomes observed as a result of the PPGTP project(s)
Figure 6: Perception of industry representatives on indirect or unintended outcomes observed as a result of the PPGTP project(s). Responses in the ten categories are presented as a horizontal bar graph starting with the highest percentage to the lowest percentage.
- Increased competitiveness: 58%
- Better stakeholder engagement and relationships: 43%
- Met or improved reponse to regulatory requirements: 32%
- Creation of new opportunities for other environmental projects: 30%
- Development / revision of internal standards and procedures: 28%
- Aesthetic benefits: 26%
- Creation of new business opportunities: 25%
- Improved health and safety of workers/community: 25%
- New or improved product, service, or technology (not in original plan): 11%
- Increased employment: 9%
Source: Survey with PPGTP industry project representatives (n=53 respondents)
Capacity building: The PPGTP resulted in an emerging ability for CFS to effectively design and deliver a large and industry-focused initiative. For example, documentary and interview evidence indicated that the PPGTP helped lay some groundwork for the NRCan Investments in Forest Industry Transformation (IFIT) Program.
IFIT was announced in Budget 2010 and will run until March 2014. It provides $100 million to encourage the next phase of mill transformation,
Supporting the research and commercialization of leading-edge technologies that will provide new applications and products for forestry companies, now that much-needed energy efficient equipment is in place to bring pulp and paper facilities back to a competitive level.Footnote 73
The IFIT application process and its Applicant Guide have built on the tools created for PPGTP, as IFIT short-listed proposals must go through a financial risk assessment and submit an Environmental Impact Statement as modeled by the PPGTP. The Guide indicates that (as for PPGTP) IFIT may conduct an Aboriginal consultation process for projects where appropriate. Quarterly, annual, and final project reporting requirements are outlined in the Guide.Footnote 74 Evidence provided from interviews and a review of the documents suggests that CFS thus adopted processes created for the PPGTP to a new program aimed at a similar stakeholder community.
The industrial stakeholders also experienced an increase in capacity: as external interviewees noted that the PPGTP process helped them cultivate knowledge on how to apply for a federal funding Program. In some cases, this knowledge was in fact used to apply for IFIT funding.
Relationship building: Most interviewees across all groups noted that over the course of the PPGTP, communication lines opened significantly. It was widely cited that before the Program, direct interaction between individual industry operations and the CFS was quite limited. As a result of the regular contact between proponents and government stakeholders during this Program, both parties felt a stronger relationship had been fostered that increased understanding of one another’s processes and needs. As indicated in the PPGTP Report on Results, as well as by a little less than half (43%) of the survey respondents and 40% of external interviewees at the project level, a similar relationship-building exercise also occurred between mills and their surrounding communities.
4.2.2 Demonstration of Efficiency and Economy (Issue 5)
The Program efficiently managed its resources with the aid of several insightful management and operational procedures. Budgeted expenditures were in line with actuals while industry and other stakeholders contributed more than $250 million to PPGTP projects, totalling $1.2 billion in direct investments. Additionally, the operation to total program costs ratio, staff size and service standards for PPGTP were comparable to similar initiatives.
However, external stakeholders suggested opportunities for administrative improvement regarding clearer reporting templates and flexibility around funding release. Finally, a lack of full-time resources or Program budget allocated after the Program closing date meant that resources had to be diverted from elsewhere in the Department to conduct closing activities.
Evidence indicated that the Program efficiently managed its resources and activities. For example, planned expenditures were in line with actual spending (see Table 1Section 2.5) and projects leveraged over $250 million dollars (see Figure 7) from proponents and other sources (such as provincial funds)Footnote 75. This represented more than 20% of PPGTP contributions. Actual Program operating costs (as a percentage of actual total program funding) were below 7% initially and decreased each year thereafter.Footnote 76 When examined in the context of the case study, this was comparable to Industry Canada’s SADI Program, which reduced its ratio from 13.8% to 4.2% in three years.Footnote 77
The case studies also indicated that the PPGTP had a relatively small staff (16 FTEs on average) compared to the other two Programs (60 or more staff for Programs of >$800 million) and that its activities were delivered in a similar timeframe to the other two Programs. For example, it was estimated that the average time from initial PPGTP proposal submission to signing of the Contribution Agreement was 14 weeks, and within the service standards set by the Program (9-16 weeks). Footnote 78 In contrast, the SADI Program had averaged 25.2 weeks from the start of the due diligence phase to obtaining departmental sign-off on CAs (with a service standard of 24 weeks),Footnote 79 while the FIPP Program was able to process its payments (no Contribution Agreements to sign) within 6 weeks.Footnote 80 This information was not provided for benchmarking purposes but rather to demonstrate that PPGTP activities were conducted with comparatively limited human resources and within a timeline that was within the range of other large-scale industry-focused initiatives.
Figure 7 PPGTP project contributions ($ Millions)Footnote 81
Figure 7 PPGTP project contributions ($ Millions)Footnote 81
|Contributor||Amount ($ millions)||%|
* Other includes provincial and other governments funding programs, in-kind resources, as well as unspecified sources of funding
Source: Program data: Leveraged Funds (October 2012)
With respect to efficient administration of Program activities (see Logic Model section 2 for examples), survey respondents had a generally positive perception of the PPGTP application process and were satisfied with the guidance and feedback provided by PPGTP staff (93%), with the promptness of funding release (80%), and with the clarity of the application guide (78%). Nonetheless, when asked about opportunities for administrative improvements, about a third of respondents suggested that performance reporting templates were difficult to use and could be better aligned to common (and therefore familiar) industrial templates, such as those used for provincial reporting. Another third indicated that flexible billing schedules and/or electronic claims submissions could be considered for future initiatives. This was confirmed by the interview data as well.
Management and operational procedures: The evaluation found that insightful management and operational procedures contributed to effective and economic use of the given resources. Several examples are worthy of mention.
The use of Memoranda of Understanding or MOUs created efficiencies by expediting or facilitating otherwise long administrative procedures (examples follow).
- Technical support for the PPGTP was contracted out by the CFS to Environment Canada’s (EC) forestry, agriculture and aquaculture division. EC duties included verification of black liquor production, project technical reviews, validation of engineering and cost estimates and monthly reporting to the CFS. The duration of the agreement was from September 21, 2009 to March 31, 2010 for a total of 0.5 FTEs (engineer and senior engineer).Footnote 82
- MOUs were signed with the NRCan Shared Services Office (SSO) for priority processing of grants and contributions in fiscal years 2010-11 and 2011-12. It paid for the services of one FTE and established service standards for processing a set number of claims received per day. This avoided project delays due to unprocessed claims.Footnote 83
- An MOU was signed with the Business Development Bank of Canada (BDC) for assistance in evaluating the financial viability of applicants pre-approved to receive PPGTP funding. This included BDC development of a Risk Assessment Report tool that NRCan could use to make a determination of whether it was prudent to release advance funds to the applicant.Footnote 84 Further, it allowed the Program to proactively allocate resources if needed and/or allowed for timely intervention in higher risk cases.
Negotiating multi-project Contribution Agreements (CAs) was an effective use of limited resources. The CA development process is resource intensive and there is a lot of overlap between CAs signed with a common proponent. Therefore, the PPGTP elected to use “multi-project CAs”, in which one CA was signed for a series of projects with the same proponent. This approach reduced the resources required by both the proponents and the Program to develop CAs. As well, time to process amendments was reduced because already approved funding was able to move between projects with a single amendment, subsequently lessening the demands on proponents and NRCan, and thereby increasing the efficiency of the Program.Footnote 85
The PPGTP CA Terms and Conditions allowed for retroactive payments. On average, the time from proposal submission to CA signing was 13 weeks. Allocated funds would have been lost without the ability to retroactively account for project costs during the time between the project approval and the CA being formally signed.Footnote 86 In addition, Program management made an early decision (June 2010) to broaden its definition of “costs incurred” to better align resources with proponent cash flow and minimize the number of companies requesting advance payments.Footnote 87
A streamlined staffing strategy was required for engaging human resources within a finite timeline considering the Program had a fixed end date. Using secondments, consultants and contact workers not only contributed to a diverse skill set but also allowed for efficient use of short-term resources.Footnote 88 Documents indicated that a total of 21 individuals were recruited over three years to deliver the Program. At Program close, all but three (two indeterminate and one term) were placed either within the CFS and other areas of the department, or had returned to the private sector.Footnote 89The resources remaining were assigned to Program close-out activities until September, 2012.
Program close-out: Close out activities included financial audits of companies requiring outside consultants; and several technical audits, final report reviews, and project claims processing to be conducted by CFS staff. Footnote 90 With respect to planning for the end of the Program, there was evidence regarding discussions of Program close-out, but no evidence indicating that a clear phase-out plan was agreed upon. Because the PPGTP terms and conditions officially expired March 31, 2012, there were no dedicated funds documented in budgetary planning for 2012-13 although wrap-up was expected be concluded in the first half of that year, and staff resources were allocated to conduct those activities until September, 2012.
A request was made to the Deputy Minister (DM) that funds from the PPGTP’s 2011-12 operating surplus directly be carried forward to 2012-13 and earmarked to fund Program close-out and final due diligence. In total, these costs were estimated at $565,000, +/- $50,000. This included approximately $150,000 in salary and benefits, and $415,000 in operating costs, the majority associated with project audits.Footnote 91 However, it is clear that funding needs to address any PPGTP program delivery requirements after March 31, 2012 have been covered through available CFS discretionary operating funds.
Question: Is performance information being collected and used on an ongoing basis?
Yes, there was clear evidence that detailed Program performance metrics were collected since the inception of the Program and would continue to be collected after Program closure. To date, this information was used both to contribute to departmental performance reports and to create the Program’s Report on Results. However, there was no clear evidence detailing how information to be provided by the industry two-years post-project would be used to inform on the longer-term impact of the Program.
Performance measurement: At the Program level, data collection templates and reporting processes were well organized and documented from Program start to finish. Examples include: the initial application form to establish proponent eligibility;Footnote 92 Excel spreadsheet templates to quantify black liquor production based on established technical literature;Footnote 93 and the development of a five-phase procedure for verification of calculated black liquor credits.Footnote 94 The Program subsequently produced and distributed a Performance Measurement Guide to enable proponents to “report accurately and consistently on environmental and ancillary benefits resulting from Program funding.”Footnote 95 A colour-coded Environmental Benefits Reporting Template was also distributed so that proponents could report data before, during, and after the project. Mandatory metrics (based on the project proposal) included:
- air emissions
- water discharge
- energy efficiency
- renewable thermal energy
- renewable electrical energy
- other environmental benefits
- annual production, and
- project online and on spec (yes/no).
The considerable quantitative evidence on achievement of outcomes presented in section 4.2.1 suggested that the data collection strategy was adequate to gather sufficient data for subsequent analysis.
Reporting and ongoing data collection: Despite the comprehensive performance monitoring process, the evaluation found a lack of clarity surrounding ongoing use of the data for future reporting or decision-making purposes. The Performance Measurement Guide stipulated that results on quantitative performance indicators “must be reported by March 31st of the two complete calendar years following project completion.” However, internal interview data indicated that this kind of post-project reporting might not be essential given that after March 31, 2012 there were no dedicated resources for the analysis of the data collected and no documented evidence of a CFS plan for future use of this information.Footnote 96 This was tied to the issue of Program close-out, as discussed in the previous section.
In addition, the Program was not designed to systematically capture data that might inform ancillary benefits or other outcomes of the PPGTP. Specifically, PPGTP funding recipients were only required to report on environmental indicators set out in their initial project objectives, as per the stated purpose of the Program. There was no requirement to report other benefits, even if they were observed. Recommended (but voluntary) reporting included: health and safety, job creation, job skills development, community and Aboriginal relations, and other socio-economic benefits. Thus, while clear quantitative data was being consistently collected, there was potentially much more that was available but not reported. With respect to the significant materiality of the PPGTP, this represented a missed opportunity to have reliable information to better understand the impact of large-scale government interventions in specific circumstances. Such information could help to guide future Departmental or GoC decisions in such situations, as well as provide the cost-benefit /effectiveness data to improve decision-making.
Question: To what extent can the outcomes measured be attributed to the PPGTP?
National-level outcomes were not distinctly addressed in this evaluation because sufficient time has not elapsed to capture longer-term impacts. At a project-level however, it was likely that about half of the projects would not have proceeded without PPGTP funding. If they did proceed, it likely would have been within a longer timeframe or reduced scope.
To answer this question, survey respondents and interviewees were asked to discuss the likelihood that their projects would have proceeded in the absence of PPGTP. Thirty-one percent of survey respondents indicated “not at all” while an additional 26% indicated that the likelihood was less than 50%. According to these respondents, probably implications of the project not proceeding included increased production/energy costs (66%), possible layoffs/closures (29%), and negative environmental impacts (22%). Similarly, less than half the external interviewees thought that projects would have proceeded in the absence of PPGTP funding as most companies simply did not have the capital to tackle large projects.
Thus the evaluation concluded that without PPGTP funding, it was likely that about half of the projects would not have proceeded. If they had been implemented, it likely would have been within a longer timeframe or reduced scope, and the environmental benefits would not have been realized during the Program (and evaluation) period.
While it is not yet possible to measure the long-term outcomes and their attribution to PPGTP, interview and survey data indicate that the improvements would continue. Three of the four company executives interviewed said that they had ongoing projects as a result of the PPGTP and the fourth said that the company was in a better economic position to consider ongoing projects. In addition, two other external interviewees said that the PPGTP provided the foundation for programs like IFIT, giving the industry the required “step change” to push for innovations/diversified revenue sources that would allow for longer term sustainability. Five out of ten external interviewees noted continuing projects as a result of the PPGTP projects. From the survey data, industry respondents indicated that after the PPGTP funded project ended, focus on reducing energy usage (14%) continued, and 7% reported a motivation to upgrade other installations /equipment when asked about the contribution of PPGTP funding to the longer-term environmental and commercial sustainability.
Yes, building on the comprehensive “lessons learned” identified via internal interviews and documents, the evaluation highlights four best practices to consider for future Programs, as observed across case studies conducted on two federal initiatives of similar size and scope.
Internal interviews and documentary evidence presented several lessons learned from the Program’s perspective. Combining this information with data gathered from the SADI and FIPP initiatives (examined in the case studies) resulted in four best practices that NRCan could consider in developing future initiatives.
Build on existing strengths, avoid old constraints: Both SADI and FIPP were based on previous Programs within their respective departments. Each Program benefited from continued federal support, strong institutional knowledge gathered over time, and clearly established relationships with the Program’s target audience. However, for both of these Programs, the reuse of some processes or tools became a constraint (e.g., SADI maintained restrictive IP policies from its predecessor, whereas for FIPP, the reuse of an old database challenged the payment processing system).
Alternatively, as the PPGTP was not confined by a previously established structure, it had the flexibility to provide a rapid response to stakeholders and to use unconventional tools (e.g., multi-project CAs). Furthermore, interviewees noted that processes pioneered for PPGTP (e.g., Evaluation Assessment guide) had been modified by CFS for use in the IFIT Program, thus taking advantage of the capacity built during PPGTP.
A combination of external factors and strong political resolve may favour Program creation: For all three Programs, it was evident that a combination of external factors strongly influenced Program genesis. The most important of these included a vocal industrial lobby (SADI, PPGTP), a rapid drop in domestic and international markets (PPGTP, FIPP), and a government disposed to address needs with targeted funding (SADI, FIPP, PPGTP). Similarly, all three Programs were supported by engaged senior-level managers. Thus, a clear political willingness to implement each Program was an apparent facilitator.
Program delivery is facilitated by a) clearly defined roles within a committee or team-based governance structure and b) the opportunity for stakeholder consultation during the planning stages: For both the SADI and PPGTP, federal policies on CAs imposed a certain committee-based governance structure. This generally comprised a financial group, a corporate group, and a series of smaller groups (e.g., claims reviews, risk management, operations, communications, IT, etc.) for Program delivery. FIPP was also organized similarly, with key functional groups reporting to a central committee. Regular communication and clearly defined roles within this structure enabled a collaborative process and smooth internal function for all three Programs.
In addition, for both FIPP and PPGTP, there was a conscious effort during the planning stages to consult with key internal and external stakeholders (i.e., those who would be delivering the Program and those considered as recipients). An upfront discussion allowed for clear identification of recipient needs and pinpointed the best delivery mechanisms/structures to meet those needs.
Program efficiency is clearly influenced by key management and operational decisions: For all three Programs, efficiency and cost-effectiveness were positively or negatively affected by some important decision-making processes or lack thereof. For example, it was found that application and claims processing time were greatly improved by:
- A comprehensive outreach strategy to the targeted recipients – PPGTP and FIPP successfully targeted and reached almost the entire relevant sector, on the other hand SADI initially had a weak outreach strategy, which was reflected by poor Program uptake.
- The use of service standards – PPGTP and SADI had clear service standards in place facilitating efficient processing of applications and claims. While FIPP did not employ service standards, the two programs that succeeded FIPP recognized this as a weakness, and implemented service standards thereafter.
It was found that all three Programs leveraged existing internal and external capacity (i.e., knowledge, ability, skills). This had the effect of streamlining processes, improving in-house expertise and facilitating an iterative improvement cycle over time.
The performance monitoring strategies for both PPGTP and SADI were well outlined, with clear indicators, targets, and regular reporting expected from recipients. In addition, there were clear mechanisms for reporting internally at the departmental level. This allowed both Programs to create a “report on results” or annual “highlights” document so that impacts could be measured and disseminated to stakeholders. Alternatively, there was little evidence of a clear performance monitoring strategy for the FIPP. Conceding that this Program was never designed as a repayable contribution, it was nonetheless difficult to quickly determine its impact several years later.
Lack of a concrete plan for Program closure was apparent for both the PPGTP and the FIPP. For both Programs this meant diverting resources (human and financial) from other areas of the organization to make final payment amendments or to gather final data. In particular, for the FIPP, this delayed timely resolution of appeals after Program closure. However this also presented an opportunity for improvement, as the Programs following FIPP implemented a distinct procedure to address post-Program appeals. For SADI, the Program is ongoing and likewise, there is similarly no defined close-out plan. Considering the experience of PPGTP and FIPP, future initiatives should seek to implement a clear and coherent plan for Program closure.
5.0 Conclusions and Recommendations
Relevance: The evaluation found that the PPGTP addressed the needs of the Canadian public and the pulp and paper industry in an evolving global market environment – first, by supporting quantifiable and targeted environmental and energy improvements and second, by maintaining the sector’s economic viability during a time of limited capital financing and international competition.
The PPGTP objectives were aligned with government priorities, and with the departmental strategic objective of Environmental Responsibility. The evaluation also found that there was a clear role for the federal government in this type of Programming considering the wider environmental benefit for Canadians and the fact that the PPGTP ran in parallel with similar targeted funding initiatives to stimulate capacity in other key industrial sectors.
Performance: The evaluation concluded that the intended outcomes of the PPGTP were clearly achieved, with good progress towards the longer-term environmental and commercial sustainability of the pulp and paper industry. While the short two-year time frame for project implementation presented a challenge both for the Program and the industry, success was facilitated by a number of key decisions made during the planning and delivery stages (e.g., non-competitive funding model, targeted outreach strategy, diversity of in-house expertise). The PPGTP also clearly resulted in enhanced capacity-building and relationship-building for both internal and external stakeholders.
Recommendation 1: Building on the linkages established through the PPGTP, the CFS should continue to develop its understanding of industry operation and maintain strong working relationships with its industry stakeholders. It was widely cited that before the Program, direct interaction between individual companies and the CFS was quite limited. As a result of the regular contact and personal interactions between proponents and Program representatives, both parties indicated that a stronger and more familiar relationship had been fostered. The main outcome of the improved relationship was that needs were better understood on both sides, and Program processes were better designed. The CFS could encourage ways to build on these informal industry linkages and seek out opportunities for ongoing collaboration. This could foster more efficient and effective communication, helping to inform the design of future Programs.
The evaluation concluded that PPGTP activities were conducted in an efficient and cost-effective manner, similar to other federal initiatives comparable in size and scope. Performance monitoring and reporting processes enabled detailed measurement of outcomes to date. Much of the cost-effectiveness of this Program was attributed to several unique management and operational procedures (e.g., use of MOUs, multi-project Contribution Agreements, retroactive payments, and streamlined staffing processes). Furthermore, when examining the PPGTP along with comparable initiatives, the evaluation identified several best practices (such as early stakeholder consultation, and a team-based governance structure) that allowed for efficient Program delivery.
Recommendation 2: NRCan (and CFS) should continue to leverage the unique elements and best practices of PPGTP design in order to capitalize on the Department’s emerging ability to effectively deliver a large, industry-focused initiative. PPGTP’s success relied upon some unique facilitators such as senior management support and stakeholder consultation early in the planning phase; an eligibility-based, flexible funding model; creative use of MOUs; the ability for retroactive payments and more. Recognizing that NRCan had already leveraged some elements of the PPGTP design for its IFIT Program, a continued focus on information management and knowledge dissemination – especially with regard to lessons learned and best practices – would ensure that departmental corporate memory was maintained and used to inform the design of future Programs.
Notwithstanding performance reporting to date, it is not clear how data collected one and two years post-project (as stipulated by the Program) will be analyzed, reported or used for future CFS decision-making, especially given the Program close-out context.
Recommendation 3: To leverage the utility of post-project data collection, CFS should consider establishing a formal means to compile and communicate the longer-term outcomes of the PPGTP. There is both a demonstrated need (from a Canadian citizen perspective) and an expectation (from an industry stakeholder perspective) to update the Program’s Report on Results once all post-project reports have been received. Such an update would provide evidence of the actual transformative impact of the PPGTP on the longer-term environmental and commercial sustainability of the pulp and paper industry. A formal plan would also justify the rationale and utility of the mandatory data reporting requirements bounded to project Contribution Agreements
Given recommendations 2 and 3 above, it would be prudent for NRCan to ensure that when designing future programs, the development of clearly defined plans for program close-out, including resource allocation for wrap-up activities, be considered. In addition, in light of the best practices identified in this evaluation (e.g., avoid old constraints, documenting decisions that affect Program efficiency), there would be value in examining any lessons learned that result from ongoing PPGTP close-out work. In the interest of documenting lessons learned, an opportunity exists in the context of this Program to learn more about the effect of the absence of a plan for closing-out a Program.
Recommendation 4: NRCan should consider documenting the PPGTP life-cycle costs and other implications for CFS to inform future design and risk assessment for grants and contributions programs. This would build on existing strengths and engender continuous improvement.
Finally, PPGTP recipients were required to report on specific environmental indicators set out in project proposals. Reporting of other benefits (e.g., worker health and safety, job creation, skills development, community/Aboriginal relations, etc.) was recommended but voluntary. The evaluation found that some of these ancillary benefits (e.g., job creation) were indeed observed but could not be conclusively quantified. In this respect, the evaluation recognizes that the decision made at Program inception to not include socio-economic indicators in the project performance reporting system was based on sound Program management practice. The Program’s core objective was environmental, not socio-economic, and voluntary reporting reduced the reporting burden for the industry. However, from the Canadian public’s perspective, the economic context, national scope, large materiality and punctual use of public funds to support the pulp and paper industry could certainly be considered as a unique public policy decision. As such, there was a missed opportunity to capture socio-economic data, to learn and report back to Canadians and the federal community.
Recommendation 5: CFS should consider the opportunity to capture key socio-economic data and report back to Canadians for future industry-focused Programs with a national scope and large materiality. Future NRCan Programs aimed towards rapid industry support or within a unique economic context should consider mandatory reporting of key socio-economic benefits in order to quantify the value of the Program to Canadians. Furthermore, collection of such information would present an advantage to the department itself in that this data could be used to develop predictive models or new innovation/transformation indicators thus continuing to inform and optimize Program design.