Evaluation of the Class Grants and Contributions Program

Table of Contents

Executive Summary

Introduction

This report is an evaluation conducted of the Class Grants and Contributions (Gs&Cs) Program. The evaluation covers $39.5 million of Natural Resources Canada (NRCan) funding for 217 class grants and 515 class contributions during the period 2006-07 to 2010-11.

Background

The Class Gs&Cs Program is a funding authority available for use by all sectors. The Program has been in existence since 1997 and the terms and conditions expire on March 31st, 2012. The objective of the Program is to support the research, development and promotion of activities that will contribute to the achievement of the Department’s goals and objectives for which there is no specific program authority (i.e., terms and conditions) but fall within the mandate of NRCan.

Some examples of expected outcomesFootnote 1 for the Program include:

  • increased knowledge in natural resources policy and priority areas;
  • transfer of technologies to stakeholders and industry groups; and
  • increased awareness of fields of study and careers in the Canadian natural resource industry sectors.

The Class Gs&Cs Program allows NRCan to take advantage of opportunities that are a cost-effective and efficient means to achieve the expected results of NRCan’s mandate, opportunities that are not specifically covered under other NRCan grants and contributions programs.

From 2006-07 to 2010-11, total departmental expenditures on class Gs&Cs were approximately $39.5 million, 82 percent of which was class contributions and the remaining 18 percent class grants. The average value of a contribution for that time period was $63,000, and the average grant was $32,500. There is no overall budget cap for the Program. Sixty-one percent of all grants and 37% of all contributions were less than $25,000.

Class Gs&Cs generally fell into two categories. Sixty-five percent of all expenditures fell under research and development. The balance of expenditures covered a very wide range of activities ranging from support for students to attend conferences to publications. Recipients included individuals, non-governmental organizations, academic institutions, industry and governments.

In addition to amounts allocated for the Class Gs&Cs Program, by sectors, funding may be augmented by reallocations from other sources of funds. The $39.5 million in expenditures during the evaluation period includes $31.5 million in expenditures from existing reference levels/reallocations and $8 million in expenditures related to specific programs (e.g., Geoscience for Energy and Minerals) where the decision was made to use the terms and conditions of the Class Gs&Cs Program.

During the evaluated period, the Innovation and Energy Technology Sector managed 38 percent of the class Gs&Cs expenditures, while the Canadian Forest Sector, Energy Sector, and Earth Sciences Sector managed 20, 18 and 11 percent respectively. Academic organizations (31 percent), industry (28 percent), and non-governmental organizations (22 percent) received the largest amounts of grants and contributions support.

With respect to governance, the Results-based Management and Accountability Framework for the Program stipulates that under the Assistant Deputy Minister of the Corporate Management and Services Sector (CMSS), the Director General, Financial Management Branch has lead responsibility for establishing and maintaining the T.B. Authority for the Class Gs&Cs Program. However, responsibility for planning, managing and administering of the program is shared. At an operational level, lead responsibility may be delegated by the assistant deputy ministers to director generals, or in the case of regions, to regional directors.

The Centre of Expertise on Grants and Contributions (COE), Financial Management Branch, was established in January of 2004 with a mandate to provide advice and direction to all sectors and regions on Gs&Cs issues.Footnote 2 The COE is responsible for, among other things, departmental corporate monitoring for all Gs&Cs programs and development of various tools, guides and policies, and training for managers in the National Capital Region (NCR) and regions.Footnote 3 The COE conducts its work based on direction provided by the Transfer Payment Review Committee (TPRC).

It should be noted that the evaluation findings do not reflect the impact of the accountability and management changes made in response to the 2011 internal audit. Most of these changes occurred after the end of the evaluation period, March 31, 2011. For example, a Class Gs&Cs Management Framework was approved by NRCan’s Transfer Payment Review Committee in July 2011. This framework is designed to clarify the management and accountability structure and strengthen monitoring and reporting requirements. However, the evaluation recommendations take into consideration how the measures put in place following the audit can be further strengthened.

Findings

Relevance

Continued Need

Evidence collected for this evaluation indicates that there is a continued need for the Class Gs&Cs Program. Its broad scope provides the flexibility to allow all sectors to support third party promotion and research and development (R&D) activities in response to emerging priorities, periods of transition, and program needs when no other appropriate transfer payment authority exists.

Annual Class Gs&Cs Program expenditures have increased from $4.8 million in 2006-07 to $13.7 million in 2010-11, with the most significant increases in funding, particularly for R&D activities, occurring from 2008-09 to 2010-11. Part of the reason for this increase is attributed to the sunsetting of the Energy Efficiency and Alternative Energy Program in 2008-09.

Alignment with Government Priorities and NRCan Strategic Objectives

The evaluation found that the class Gs&Cs are aligned with a range of Government of Canada and departmental priorities, particularly those relating to clean energy, economic competitiveness, and innovation. However, alignment with NRCan priorities and strategic outcomes should be more clearly articulated in project documentation.

Alignment with Federal Roles and Responsibilities

The federal government has a legitimate role in the activities undertaken through the Class Gs&Cs Program given the supporting legislation and mandate. NRCan plays an appropriate coordination and leadership role in the R&D and promotion activities supported by the Program. NRCan is well-positioned to play this role given its knowledge of the natural resource sectors.

Performance

Achievement of Expected Outcomes

The Class Grants and Contributions Program supports NRCan’s strategic outcomesFootnote 4, particularly environmental responsibility and economic competitiveness, when no other appropriate mechanisms exist. The evidence suggests that it contributes to these outcomes by leveraging expertise and resources and building the natural resource sectors’ capacities.

Given the diverse projects supported by the Class Gs&Cs Program, it was challenging to assess overall program performance. However, the evaluation evidence suggests that the Program has contributed to:

  • increased interest with respect to priority areas or energy efficiency and sustainable development activities;
  • increased knowledge in natural resources policy and priority areas or natural resource industry sectors; and
  • new or improved technologies.

While there are examples of achievement related to other specific outcomes such as “increased awareness of fields of study and careers”,“policy development” and “technology transfer,” the extent to which the Program has contributed to these outcomes is not clear given the available evidence.

Demonstration of Efficiency and Economy

The evaluation found aspects of the Program’s design and delivery to be efficient and economic, but that accountability needs to be strengthened through improved oversight and clarification of roles and responsibilities (e.g., need to clarify decision-making responsibility and process for using the Class Authority in lieu of seeking a unique Treasury Board Authority);

The following are elements of the Program’s design and delivery that support efficiency and economy:

  • reduced need for creating numerous terms and conditions for funding requests (this also facilitated timely ability to respond to emerging priorities and opportunities);
  • multiple-year funding contributed to efficiency and effectiveness;
  • completion of project outputs and milestones in a timely manner; and
  • the Program leverages funds and in-kind contributions from other sources.Footnote 5

In addition to the need for strengthened accountability, the evaluation found that there is room for improvement in terms of:

  • performance measurement (e.g., streamline logic model and enhance existing project reporting templates to ensure alignment with appropriate outputs, outcomes and the Program Activity Architecture);
  • monitoring and reporting (e.g., enhance reporting to the Transfer Payments Review Committee by including all expenditures using the Class Authority);
  • timeliness of the approvals process;
  • communication between the Centre of Expertise and the sectors; and
  • development and delivery of tools and training to enhance understanding and efficient use of class Gs&Cs.

Conclusion

In conclusion, the Class Grants and Contributions Program is relevant and supports NRCan’s strategic outcomes, particularly with respect to economic competitiveness and environmental responsibility. Its broad scope facilitates the use of Gs&Cs across the Department, allowing managers to respond to emerging opportunities and priorities. Its use has also supported existing programs when there is no other appropriate G&C authority.

However, the broad nature of the Program and shared responsibility for delivery also creates challenges in terms of governance, accountability and performance measurement. The Centre of Expertise implemented a number of changes in 2011 to improve management and accountability. The evaluation found that the relevance and performance of the Program could be enhanced by further strengthening these changes to ensure clarity of roles and responsibilities, improved performance measurement, monitoring and reporting, and the continued implementation of a communications and training strategy.

Recommendations and Management Response

Recommendations Management Response Responsible Official/Sector (Target Date)
NRCan’s Centre of Expertise (COE) on Grants and Contributions should further strengthen the management and accountability framework of the Class Grants and Contributions Program by:
1. Strengthening the governance of the Class Gs&Cs Program to support regular monitoring and reporting, decision-making responsibility and approvals. Accepted.

CMSS - FMB/COE, as reflected in the Class Gs&Cs Management Framework, will generate and provide a class Gs&Cs report semi-annually (mid-year and year-end report) to the Transfer Payments Review Committee, which includes expenditures of all agreements under the Class Gs&Cs Authority

FMB/COE will also further strengthen the Framework by:
  • adding project approval thresholds at the DM and ministerial level;
  • revising the Delegation of Financial Signing Authorities Instrument to include the new project approval thresholds;
  • updating the Class Gs&Cs Management Tools to capture information with regard to the applicable Program Activity Architecture sub-activity code and funding in order to link agreements to departmental objectives and clarify budgetary considerations;
  • updating the Roles and Responsibilities section to reflect the area(s) responsible for decision making; and
  • updating the Procedures and Guidelines section of the Framework to incorporate all recent changes.
CFO/ADM, CMSS

June 2012
2. Improving the performance measurement strategy (e.g., streamlining the logic model; modifying the reporting templates to align with the logic model and the Program Activity Architecture; and enhancing the Gs&Cs module in the financial management system). Accepted.

CMSS – FMB/COE, in collaboration with the Strategic Evaluation Division, will update and streamline the logic model to provide a better link between funded projects and the Program Activity Architecture. Further, the recording of key information within the Application for Modules and Interfaces (AMI) will be made mandatory to facilitate reporting and monitoring.
CFO/ADM, CMSS

June 2012
3. Continuing to implement the training/ communications strategy for all staff involved with the Class Gs&Cs Program (e.g., provide regular notifications of changes and updates; develop and deploy training and user-friendly tools re: process, purpose, conditions of use, and reporting requirements). Accepted.

CMSS – FMB/COE will continue to provide notifications of changes and updates as needed via e-mail, bulletins and/or information pieces posted on the Source. User-friendly tools about process, purpose, conditions of use, and reporting have been developed and deployed via the Intranet as components of the Class Gs&Cs Management Framework (specifically, the Class Gs&Cs Management Tools)

Information specific to class Gs&Cs is currently being added to COE’s online Training Modules and COE’s training materials for in-class training.
CFO/ADM, CMSS

Ongoing

June 2012

1.0 Introduction and Background

1.1 Introduction

This report summarizes the findings of an evaluation conducted on the Class Grants and Contributions (Gs&Cs) Program. The evaluation covers $39.5 million of NRCan funding for 217 class grants and 515 class contributions for the period 2006-07 to 2010-11.

1.2 Background

The Class Gs&CsFootnote 6 Program is a funding authority available to NRCan managers in all sectors. The Program has been in existence since 1997 and the terms and conditions expire on March 31st, 2012.

The broad objective of the Program is to support the research, development and promotion of activities that will contribute to the achievement of the Department’s goals and objectives for which there is no specific program authority.Footnote 7

Some examples of immediate outcomesFootnote 8 for the Program include:

  • increased knowledge in natural resources policy and priority areas;
  • transfer of technologies to stakeholders and industry groups; and
  • increased awareness of fields of study and careers in the Canadian natural resource industry sectors.

The Class Gs&Cs Program allows NRCan to take advantage of opportunities that are a cost-effective and efficient means to achieve the expected results of NRCan’s mandate, opportunities that are not specifically covered under other NRCan grants and contributions programs.

The Class Gs&Cs Program allows NRCan to fund initiatives that do not fall within other existing transfer payment authorities. Traditional grants and contributions programs provide support to specific program areas or initiatives. However, the “class management” approach supports grants and contributions that are designated to a broad theme. According to the terms and conditions, the Class Gs&Cs Program generally supports the following NRCan program activity areas:

  • Economic Opportunities for Natural Resources;
  • Natural Resource-based Communities;
  • Clean Energy;
  • Ecosystem Risk Management;
  • Adapting to a Canadian Climate and Hazard Risk Management; and
  • Natural Resource and Landmass Knowledge for Canadians.

Examples of activities that can be funded under the Class Gs&Cs Program include:

  • research papers related to new technology or advancement in sciences related to natural resources;
  • development or enhancement of new technology;
  • conferences/workshops which promote areas of interest in natural resources or which have R&D themes;
  • organizations involved in promoting natural resource activities;
  • students in a natural resource field of study; and
  • the dissemination and distribution of publications.

Eligible recipients for class Gs&Cs include:

  • individuals;
  • Canadian profit and non-profit organizations;
  • foreign/international non-profit and for-profit organizations;
  • other levels of foreign governments;
  • Canadian and foreign academic institutions;
  • industry and research associations; and
  • provincial, territorial, municipal and regional governments.

The maximum amount payable under a class grant is $250,000 per fiscal year, and $750,000 per fiscal year for a class contribution. Grant and contribution agreements may be one- or multi-year.

1.2.1 Program Expenditures

From 2006-07 to 2010-11, total departmental expenditures on class Gs&Cs were approximately $39.5 million, 82 percent of which was class contributions, and the remaining 18 percent were class grants.

In addition to its own allocated budget, the Class Gs&Cs Program is typically augmented by reallocations from other sources of funds. The $39.5 million includes $31.5 million in expenditures from existing reference levels/reallocations and $8 millionFootnote 9 in expenditures related to specific programs where the decision was made to use the terms and conditions of the Class Gs&Cs Program. There is no overall budget cap for the Program.

Annual commitments for the Class Gs&Cs Program are set by assistant deputy ministers, in consultation with headquarters and regions, based on previous year’s use and commitments (e.g., multi-year agreements). In the event that new opportunities or priorities arise and there is no class G&C allocation available, then the required resources may be obtained via inter-program transfers within or between sectors.Footnote 10 Sectors manage these exchanges and reallocations through their Sector Financial Advisor’s Office.

The average value of a contribution agreement between 2006-07 and 2010-11 was $63,000, whereas the average grant agreement was $32,500. As Table 1 below shows, the value of the average contribution agreement rose steadily from $29,800 in 2006-07 to $108, 500 in 2010-11. The average value of grants fluctuated between an average low of around $25,000 in 2006-07 to $39,000 in 2010-11. With the exception of 2007-08, annual expenditures on class Gs&Cs has increased each year over the evaluation period, going from about $4.8M in 2006-07 to $13.7M in 2010-11.

Table 1: Class Grants and Contributions Expenditures, 2006-07 to 2010-11
FY # Class Grants $ Class Grants Average $/Grant # Class Contrib. $ Class
Contrib.
Average $/Contrib. Total #
Class Gs&Cs
Total $
Class Gs&Cs
2006-07 29 726,086 25,037 138 4,105,983 29,754 167 4,832,069
2007-08 28 708,544 25,305 42 1,491,275 35,507 70 2,199,819
2008-09 45 1,783,824 39,641 86 4,866,954 56,592 131 6,650,778
2009-10 66 1,907,322 28,899 140 10,163,057 72,593 206 12,070,379
2010-11 49 1,923,389 39,253 109 11,823,860 108,476 158 13,747,249
Total 217 7,049,165 32,485 515 32,451,129 63,012 732 39,500,294

Source: NRCan database – Government Financial System User-Friendly (GUFI) Module.

Figure 1 shows the distribution of expenditures on class Gs&Cs during the evaluation period by sector. The Innovation and Energy Technology Sector (IETS) managed 38 percent of the class Gs&Cs expenditures, while the Canadian Forest Sector (CFS), Energy Sector (ES), and Earth Sciences Sector (ESS) managed 20, 18 and 11 percent respectively. Together these four sectors accounted for almost 90 percent of departmental class Gs&Cs expenditures during the evaluation period.

Figure 1: Distribution of Class Gs&Cs Expenditures by Sector 2006-07 to 2010-11

Figure 1: Distribution of Class Gs&Cs Expenditures by Sector 2006-07 to 2010-11
Text Version

Figure 1 shows the percentage of class Gs&Cs expenditures by sector over the evaluation period 2006-07 to 2010-11.  Total Program expenditures from 2006-07 to 2010-11 were $39.5 million.  The Innovation and Energy Technology Sector accounted for 38% of the total class Gs&Cs expenditures; the Canadian Forest Sector accounted for 20% of the total expenditures; the Energy Sector accounted for 18% of the total expenditures; ant the Earth Sciences Sector accounted for 11%. The Science and Policy Integration Sector accounted for 6%; the Minerals and Metals Sector 5%; and the Major Projects Management Office accounted for 1% of total Program expenditures. Other sectors accounted for the remaining 1% of expenditures.

 

Figure 2 shows that more than 90 percent of all class Gs&Cs over the evaluation period were less than $150,000. Almost half were less than $25,000. Eighty-six percent of the agreements over $150,000 were contribution agreements.

In comparing the distribution of class grants and class contributions separately, a higher portion of expenditures for class Grants (61 percent) were under $25,000 in comparison to contributions (37 percent). For class grants, 21 percent were in the $25,000 to $50,000 range, and 18 percent were over $50,000. For class contributions, 21 percent were in the $25,000 to $50,000 range, and 42 percent were over $50,000.

Figure 2: Distribution of Class Gs&Cs by Funding Range 2006-07 to 2010-11

Figure 2: Distribution of Class Gs&Cs by Funding Range 2006-07 to 2010-11
Text Version

Figure 2 shows the percentage of class Gs&Cs agreements by funding range from 2006-07 to 2010-11.  The total number of grant and contribution agreements from 2006-07 to 2010-11 was 732.   Forty-five percent (45%) of all agreements were less than $25,000.  Twenty-seven percent (27%) of all agreements were between $50,000 and $149,900.  Twenty-one percent (21%) of all agreements were between $25,000 and $49,900.  Seven percent (7%) of all agreements were greater than $150,000.

 

Figure 3 shows the distribution of class Gs&Cs expenditures by type of recipient. Academic organizations (31 percent) and industry (28 percent) received the largest amounts of support. Non-government organizations (NGOs) received 22 percent of the expenditures, while international organizations received 12 percent.

International (40 percent) and NGOs (35 percent) received the largest percentages of class grant funding. For class contributions, academia (35 percent), industry (32 percent) and NGOs (20 percent) received the largest portions of funding.

Figure 3: Expenditures on Class Gs&Cs by Recipient Type 2006-07 to 2010-11

Figure 3: Expenditures on Class Gs&Cs by Recipient Type 2006-07 to 2010-11
Text Version

Figure 3 shows the percentage of class Gs&Cs expenditures by type of recipient for the evaluation period 2006-07 to 2010-11.  Total Program expenditures from 2006-07 to 2010-11 were $39.5 million.  Academic organizations received 30.6% of total Program expenditures. Industry received 28.0% of total Program expenditures. NGO’s received 22.6% of total expenditures and international organizations 11.7%.   Government organizations received 3.7% of total expenditures, aboriginal groups 2.1%, utilities 1% and individuals 0.1% of total Program expenditures.

 

For this evaluation, projects were classified as R&D or promotion based on the project descriptions. Promotion includes all non-R&D projects covering a wide range of activities. Figure 4 shows that expenditures on R&D projects accounted for 65 percent of class Gs&Cs expenditures between 2006-07 and 2010-11.Footnote 11 While there were slightly more promotional projects (n=380) than R&D projects (n=338) over the evaluation period, more was spent on R&D projects ($25.6 million) than promotional projects ($13.5 million), reflecting a higher average value for R&D projects.

Figure 4: Percentage of Class Gs&Cs Expenditures by Promotion and R&D Activities 2006-07 to 2010-11

Figure 4: Percentage of Class Gs&Cs Expenditures by Promotion and R&D Activities 2006-07 to 2010-11
Text Version

Figure 4 shows the percentage of class Gs&Cs expenditures on research and development (R&D) and promotional activities for the evaluation period 2006-07 to 2010-11.  Total expenditures on class Gs&Cs from 2006-07 to 2010-11 were $39.1 million.  R&D projects accounted for 65% of total expenditures and promotional projects accounted for the remaining 35%.

 

Initial program documentation (2008) estimates that the annual cost of managing this program is $250,000 for salary and operating budget. It was also estimated that about 20 employees across various NRCan sectors are involved part-time in the process amounting to about two full-time equivalents.

1.2.2 Governance Structure

As with all Gs&Cs programs, the Class Gs&Cs Program must adhere to the applicable requirements of the Treasury Board of Canada Policy on Transfer Payments and the Financial Administration Act.

According to the 2008 RMAF, under the Assistant Deputy Minister, Corporate Management and Services Sector (CMSS), the Director General, Financial Management Branch has lead responsibility for establishing and maintaining the T.B. Authority for the Class Gs&Cs Program. However, responsibility for planning, managing and administering the Program is shared. At an operational level, lead responsibility may be delegated by the assistant deputy ministers to director generals, or in the case of regions, to regional directors.

The Centre of Expertise on Grants and Contributions (COE), Financial Management Branch, was established in January of 2004 with a mandate to provide advice and direction to all sectors and regions on Gs&Cs issues. The COE is responsible for, among other things, departmental corporate monitoring for all Gs&Cs programs and development of various tools, guides and policies, and training for managers in the NCR and regions.Footnote 12 The COE conducts its work based on direction provided by the Transfer Payment Review Committee (TPRC).

Contributions less than $200,000 and grants less than $100,000 can be approved by directors. Contributions or grants less than $500,000 require the approval of the director general. All agreements equal to or greater than $100,000 (including amendments) must be reviewed by the Centre of Expertise and Legal Services prior to the final approval by the delegated authority.Footnote 13 The TPRC reviews certain departmental grants and contributions including some class Gs&Cs.Footnote 14

2011 Class Grants and Contributions Management Framework

A 2011 internal horizontal audit of transfer paymentsFootnote 15 found that there was a need to improve the monitoring of the Class Gs&Cs Program. The audit recommended that the Centre of Expertise develop and implement a monitoring and reporting framework for the Class Grants and Contributions Program.

In response to the audit, a Class Gs&Cs Management Framework was approved by the TPRC in July 2011. This framework is designed to clarify the management and accountability structure. Highlights of the framework include:

  • the Chief Financial Officer/Assistant Deputy Minister (ADM), Corporate Management Services Sector provides financial leadership and guidance at department level, responsible for program authority maintenance and renewal;
  • program sector ADMs are responsible for use of the Program, provide overall leadership and program support to promote the appropriate use of class Gs&Cs, management of the issues regarding proponent and agreement administration; and
  • the Transfer Payments Review Committee provides general governance and oversight of transfer payment programming as per Committee Terms of Reference, to include the review of a semi-annual report regarding the use of class Gs&Cs.

2.0 Evaluation Approach and Methodology

2.1 Evaluation Scope and Objectives

This evaluation covered departmental spending on the class Gs&Cs Program of $39.5 million from 2006-07 to 2010-11.

Evaluation Issues
The evaluation covered the issues related to relevance and performance in accordance with Treasury Board Evaluation Policy. Relevance is examined in terms of the Program’s alignment with government priorities and NRCan strategic objectives; continuing need for the Program; and appropriateness of federal role and responsibilities. Performance is assessed in terms of effectiveness (achievement of expected outcomes), and efficiency and economy.

The scope and evaluation methods were calibrated according to program risk. For example, materiality and past audits were reviewed to provide guidance as to the appropriate scope of the evaluation. Consequently, the file review and case studies include a greater proportion of larger contributions.

2.2 Evaluation Methods

Multiple lines of evidence were used to assess the relevance and performance of the Class Grants and Contributions Program. These include the following data collection methods:

  • Document review: the evaluation reviewed about 20 NRCan documents to gather contextual information about the Program and for use as a data source for evaluation questions. Examples of reviewed documents include the Treasury Board Submission and Results-based Management and Accountability Framework (RMAF); audit reports (2002 and 2011); Program Activity Architecture (PAA) and Performance Measuremnt Framework (PMF); Reports on Plans and Priorities, and Departmental Performance Reports. Additionally, three external evaluations of class Gs&Cs programs delivered by three other departments were reviewed for applicable best practices and lessons learned.
  • File Review: Files of 35 class contributions and seven grants were reviewed to inform evaluation questions. Project files were drawn from a random sample of agreements stratified by agreement type, fiscal year, sector and funding amounts. More projects were selected for contribution agreements than grants, for projects with large funding amounts over small amounts, and from sectors that are the heaviest users of the Program. Of the 42 files reviewed, 14 were from IETS, eight each from CFS and ES, five from ESS, three each from MMS and SPI, and one from MPMO.
  • Database Review: A review of all class grants and contributions (2006-07 to 2010-11) contained in the GUFI Agreements Module database was conducted to obtain additional output information (217 grants and 515 contributions).
  • Interviews: In total, 28 interviews were conducted with representatives of the following groups: directors and program managers/staff from seven NRCan sectors that use class Gs&Cs (20), NRCan Financial Management Branch representatives (6), and Environment Canada representatives (2) in order to identify examples of best practices/lessons learned from other government departments (OGDs).
  • Case studies: nine contribution agreements were selected for case studies to inform relevance and performance issues at the project level. About half of the case studies were selected based on suggestions from sector representatives as examples of projects that would highlight best practices. The remaining cases were selected from the database using risk-based selection criteria (e.g., materiality, project type (R&D or promotional), and type of recipient. The distribution of case studies across sectors was IETS (3); CFS (3), ES (2); and ESS (1). Case studies involved a review of project documentation, an interview with the NRCan project manager/officer, and an interview with the proponent. (See Appendix A for a list of the case studies and reasons for selection.)

2.3 Evaluation Limitations and Mitigation Strategies

Given the large number of diverse projects supported by the Class Grants and Contributions Program, it was challenging to assess the overall program performance. In addition, files generally contained output rather than outcome informationFootnote 16 and the quality of the performance information was not consistent.

To mitigate these limitations, the evaluation utilized multiple methods to obtain both breadth (42 files - randomly selected, and database review) and depth of information (e.g., interviews and case studies).

The evaluation conducted interviews with ten funding recipients which does not provide extensive coverage of this stakeholder group. This limitation was somewhat mitigated by a file review which contained activity and progress reports from funding recipients.

3.0 Evaluation Findings

3.1 Relevance

3.1.1 Continued Need for Program

Evaluation Question Methods Assessment
Is there a continued need for the Program? Interviews; case studies; file review; document review. Yes, there is a continued need for the Program.

Summary:

Evidence collected for this evaluation suggests that there is a continued need for the Class Gs&Cs Program. Its broad scope provides the flexibility to allow all sectors to support third party promotion and R&D activities in response to emerging priorities, periods of transition, and program needs when no other appropriate transfer payment authority exists.

The need for the Program is demonstrated by the fact that expenditures for class Gs&Cs have increased from $4.8 million in 2006-07 to $13.7 million in 2010-11, with R&D expenditures increasing from $2.3 million to $9.9 million over the same time period. Part of the reason for the increase is the sunsetting of the Energy Efficiency and Alternative Energy Program in 2008-09.

The broad scope of class Gs&Cs facilitates its use by all NRCan sectors for a variety of purposes, activities and funding recipients. The Class Gs&Cs Program is generally viewed by program managers as a useful funding mechanism to meet emerging or program needs when other mechanisms such as contracting or other program Gs&Cs are not appropriate or do not exist.

There is a general consensus among NRCan interviewees, that the Class Gs&Cs Program allows the Department to take advantage of new opportunities, emerging priorities and to continue activities during transition periods. For example, the Program was used to support carbon capture and storage projects including a carbon dioxide (CO2) storage project while awaiting the creation of the Clean Energy Fund. In addition, class Gs&Cs helped the Energy Sector further its work related to integrated energy systems and to take advantage of new opportunities for market transformation activities.

File and case study evidence show that class Gs&Cs are used to support third parties in achieving results that benefit key stakeholders in the natural resource sectors. This is consistent with the objective of the Policy on Transfer Payments that transfer payment programs be citizen- and recipient-focused and are designed and delivered to address government priorities in achieving results for Canadians.Footnote 17 A review of the 42 files shows that, in the majority of cases, the files articulate a rationale for the agreement in relation to recipients or natural resources sectors. There is no file, case study, or interview evidence that the Department directly benefited from the agreements by acquiring goods, services or assets.Footnote 18

According to interviewees, the Class Gs&Cs Program is used when no other G&C authority exists or is appropriate. Some sectors such as Science and Policy Integration and the Major Projects Management Office do not have access to any program G&C authorities and occasionally use the Class Authority to further sector and departmental priorities. Sectors with access to other G&C authorities – such as the Canadian Forest Service – regularly use the Program to meet unique funding requests to support workshops, conferences and research projects which provide broad benefits to the Canadian forest industry but do not fit the more specific objectives within existing authorities.

The Class Authority (i.e., terms and conditions) is also used to deliver Gs&Cs of some programs/funding authorities (e.g., Geo-mapping for Energy and Minerals (GEM), Program for Energy Research and Development) for which there are no separate terms and conditions (Ts&Cs). Because the Gs&Cs allotments for programs such as GEM are relatively smallFootnote 19, it is considered more efficient to use the Class Authority rather than create a new set of Ts&Cs. However, interviewees note that when there are different renewal cycles for a program and for class grants and contributions, this creates challenges. For example, negotiation of multi-year agreements, which fall within the program timeframe, can be problematic as the Class Gs&Cs Authority nears the end of its cycle.

Research and Development

The Class Gs&Cs Program supports third party research and development activities which include research papers related to new technology or advancement in sciences related to natural resources or the sustainability of natural resource sectorsFootnote 20, and the development or enhancement of new technology.Footnote 21 NRCan’s support of R&D projects through class Gs&Cs is viewed by external stakeholders as filling a critical niche just prior to commercialization in encouraging small and medium enterprises’ R&D capacity, particularly in the context of the current economic climate and the scarcity of venture capital. NRCan interviewees view the Class Gs&Cs Program as appropriate for supporting emerging R&D opportunities and for supporting a wide range of R&D activities.

The use of class Gs&Cs in supporting R&D activities has increased from $2.3 million in 2006-07 to $9.9 million in 2010-11, with the most significant increases in funding occurring from 2008-09 to 2011-12. A key reason for this increase is the sunsetting of the regular Energy Efficiency and Alternative Energy Program (EEAE) in 2008-09. The EEAE Program was used by several branches within the Energy Sector and by IETS.

Promotion

The Class Gs&Cs Program also provides the flexibility to support a diverse range of promotion activities which include forums for discussion on priority areas related to natural resourcesFootnote 22 ; conferences or workshops which address areas of interest in natural resourcesFootnote 23 ; organizations involved in promoting natural resource activitiesFootnote 24 ; students in a natural resource field of studyFootnote 25 ; and the dissemination and distribution of publications.Footnote 26 These activities are considered important for maintaining the visibility of priority areas in natural resources, helping organizations to address emerging opportunities and priorities in the natural resource sectors, contributing to market transformation for various technologies related to natural resources, and informing NRCan policy decisions and discussions. Such activities also facilitate consultation, government engagement, and greater interest of youth in natural resource sectors.

3.1.2 Alignment with Government Priorities and NRCan Strategic Objectives

Evaluation Question Methods Assessment
To what extent does the Class Gs&Cs Program support projects that are consistent with departmental and government priorities? Interviews; case studies; file review; document review. The Program is aligned with a range of federal and departmental priorities.

Summary:

The evaluation found that class Gs&Cs projects are generally aligned with a range of Government of Canada and departmental priorities, particularly those relating to clean energy, economic development, and innovation. However, alignment with departmental priorities and strategic outcomes needs to be more clearly articulated in project documentation.

Alignment with Government of Canada Priorities
The evidence suggests general alignment of class Gs&Cs with a range of Government of Canada priorities. Interview, file, and case studyFootnote 27 evidence indicate that the class grants and contributions support government priorities particularly with respect to innovation, economic development and competitiveness, clean energy; and enhancing mining and forestry sectors.Footnote 28

Alignment with Departmental Priorities
A review of the files (n=42) indicates alignment with departmental priorities related to addressing climate change or furthering the clean energy agendaFootnote 29 as almost half of the agreements were judged to be aligned with this priority. Almost one-third of the files were assessed as being congruent with the departmental priority concerning the advancement of Canada’s interests and sustainability efforts globally; and approximately another third were considered to be aligned with enhancing the competitiveness of Canada’s forest sectorFootnote 30 or strengthening the minerals and metals industry.Footnote 31

While evidence suggests that class Gs&Cs align with departmental priorities and strategic outcomes, these linkages should be more clearly articulated in project documentation. The file review showed that projects were generally aligned with NRCan strategic outcomes; however, this alignment was clearly articulated in only 43 percent of the files. For the remainder of the files, alignment with NRCan strategic outcomes was not explicitly articulated but could be inferred based on descriptions of project activities, expected results, and benefits to Canadians.

There are some controls and processes in place to provide assurance of alignment of agreements with departmental priorities and strategic outcomes. Various approval checklists were evident in the files reviewed, providing varying levels of detail regarding project alignment to department priorities, but these were not used consistently. Most interviewees reported that proposals are reviewed against directorate / branch / sector / departmental priorities.

In those cases where class is used to deliver the G&C components of some programs (e.g., PERD and GEM), these projects are selected through formal project selection processes. Program planning ensures linkages to the Program Activity Architecture and its expected results, providing additional assurance of linkages to departmental priorities and strategic outcomes.

In 2010-11, the Energy Sector and the Earth Sciences Sectors conducted reviews of their Gs&Cs to ensure their alignment to departmental priorities. Following this review, some class projects were reduced or not renewed, in line with a more strategic use of funds.

3.1.3 Legitimate, Appropriate and Necessary Federal Role

Evaluation Question Methods Assessment
Is there a legitimate, appropriate and necessary role for the federal government in the Class Grants and Contributions Program? Interviews; case studies; file review; document review. The Class Grants and Contributions Program is aligned with federal roles and responsibilities.

Summary:

The federal government has a legitimate role in the activities undertaken through the Class Gs&Cs Program given the supporting legislation and mandate.

NRCan plays an appropriate coordination and leadership role in the R&D and promotion activities supported by the Program. NRCan is well-positioned to play this role given its knowledge of the natural resource sectors.

The evidence suggests that the broad scope of the Class Gs&Cs Program is useful for addressing new opportunities and programs when it is deemed inefficient to develop unique Ts&Cs to distribute those funds.

The federal government, and particularly NRCan, has a legitimate role in the activities undertaken through the Class Gs&Cs Program that supports the mandate of NRCan and are carried out under the authorities provided by the Department of Natural Resources Act, the Resources and Technical Surveys Act, the Energy Efficiency Act, and the Forestry Act.

Class Gs&Cs reviewed for this evaluation – as part of the file review and case studies – indicate that a broad range of projects have been supported that align with the mandate and requirements of the Minister as laid out in the 1994 Department of Natural Resources Act.Footnote 32 The Minister of Natural Resources is responsible, among other things, for:

  • assisting in the development and promotion of Canadian scientific and technological capabilities;
  • seeking to enhance the responsible development and use of Canada’s natural resources and the competitiveness of Canada’s natural resources products;
  • participating in the enhancement and promotion of market access for Canada’s natural resources products and technical surveys industries, both domestically and internationally;
  • promoting cooperation with the governments of the provinces and with non-governmental organizations in Canada, and participate in the promotion of cooperation with the governments of other countries and with international organizations; and
  • gathering, compiling, analyzing, coordinating and disseminating information respecting scientific, technological, economic, industrial, managerial, marketing and related activities and developments affecting Canada’s natural resources.

The 1992 Energy Efficiency Act provides the Minister of Natural Resources with the authority to conduct research activities, provide information and grants and contributions, develop partnerships, and carry out other activities and programs for the purpose of promoting energy efficiency. One of the expected outcomes of the Class Gs&Cs Program is increased interest in energy efficiency and sustainable development activities. Approximately one-third of the 42 reviewed files support energy efficiency activities.

The federal government and NRCan’s roles are considered appropriate because:

  • According to internal and external interviewees, the federal government and NRCan play an appropriate coordinating and leadership role in the activities supported through the Class Gs&Cs Program. For example, the case study of the Canadian Interagency Forest Fire Centre highlights the importance of the federal government’s ability to facilitate coordination and communication among provinces and stakeholders in order to share fire fighting resources across provinces and internationally.
  • NRCan’s enabling role in supporting R&D activities through class Gs&Cs is consistent with the federal government’s science and technology (S&T) strategyFootnote 33 in that many of the R&D agreements reviewed for this evaluation aim to build a stronger private-sector commitment to S&T, to strengthen the S&T knowledge base, and support progress towards commercialization.
  • NRCan is well-positioned to deliver the Class Gs&Cs Program given its knowledge of key stakeholders in the natural resource sectors and capacity to bring these stakeholders together. Moreover, according to internal and external stakeholders, the technical, scientific, and project management expertise of NRCan staff is necessary for the effective selection, management and monitoring of the agreements.

While NRCan has a number of grant and contribution programs directed to specific objectives, the broad scope of the Class Gs&Cs Program is considered by interviewees to be useful for addressing emerging priorities or unexpected opportunities, and to support programs when it is deemed inefficient to develop unique Ts&Cs to distribute those funds.

3.2 Performance (Effectiveness)

3.2.1 Class Grants and Contributions Logic Model

The logic model contained in the Results-based Management Accountability Framework for this program outlines various outputs and outcomes associated with R&D and promotion activities.

To facilitate the reporting of outcomes achievement, the logic model has been streamlined to reduce duplication. In addition, a number of interviewees noted that the logic model, particularly for promotion activities,Footnote 34 did not adequately describe the outcomes of the Class Gs&Cs Program (e.g., outcomes related to market transformation activities).

Class Grants and Contributions Logic Model

Outputs

  • forums;
  • conferences/workshops;
  • projects or organizations funded that promote natural resources;
  • awards or bursaries;
  • publications produced and disseminated;
  • research to advance new technology or enhancements; and
  • research to advance scientific knowledge.

Shared Outcomes (R&D and Promotion)

  1. increased interest in areas of priority for the natural resource sector/energy efficiency and sustainable development activities (increased interest in energy efficiency and sustainable development applicable to R&D only);
  2. increased knowledge in natural resources policy and priority areas or natural resource sectors;
  3. increased awareness of fields of study and careers in the Canadian natural resource industry sectors; and
  4. influence policy development.

Research and Development Outcomes

  1. new / improved technologies; and
  2. transfer of technologies to stakeholders and industry groups.

The next subsection assesses the contribution of the Class Gs&Cs Program to NRCan’s strategic outcomes. Subsequently, the achievement of the intended outcomes as specified in the logic model will be assessed.

3.2.2 Contribution to Departmental Strategic Outcomes

Evaluation Question Methods Assessment
To what extent has the Class Grants and Contributions Program contributed to the achievement of departmental strategic objectives (economic competitiveness, environmental responsibility, and safety, security and stewardship)? Interviews; case studies; file review. Evidence suggests that the Program contributes to NRCan’s strategic outcomes.

Summary:

The Class Grants and Contributions Program supports strategic outcomes, particularly environmental responsibility and economic competitiveness, when no other appropriate mechanisms exist. The evidence suggests that it contributes to these outcomes by the leveraging of expertise and resources; and building the natural resource sectors’ capacities.

Interview and case study evidence suggest that if projects had not received funding, either the project likely would have been reduced in scope, delayed or would not have occurred at all.

The analysis of performance is constrained by insufficient outcome informationFootnote 35 and the inconsistent quality of output data as contained in project progress and final reports.Footnote 36 This challenge is exacerbated by the fact that the Program encompasses a wide range of activities and outcomes.

In addition, performance and financial information for the Class Gs&Cs Program was not aggregated and reported to senior management until 2011 as a result of the implementation of an improved management / accountability structure.

Given these limitations, this sub-section will include examples of support for strategic outcomes and the conditions and features of the Program considered important for facilitating NRCan’s strategic outcomes:Footnote 37

  • Economic Competitiveness;
  • Environmental Responsibility; and
  • Safety, Security and Stewardship.

The majority of agreements reviewed for this evaluation are aligned with economic competitiveness or environmental responsibility providing a necessary condition for contribution to these outcomes. The evaluation also found a few examples of class agreements that were aligned with safety, security and stewardship, such as: support for remote sensing, the Canadian Interagency Fire Fighting Centre; and dissemination of earthquake information.

Interviewees report that many projects are successful because they have engaged a wide range of stakeholders including non-government organizations, universities, and international stakeholders (e.g., U.S. and global regulators). The engagement of key stakeholders facilitates the leveraging of expertise and resources. class Gs&Cs facilitate international activities, which provide access to considerable global expertise and information for a relatively small amount of funding. The file review corroborates leveraging of funds and in-kind contributions as most of the agreements reviewed (34 of 42) indicate leveraging.

Interview and case study evidence highlight a number of examples of class agreements facilitating economic competitiveness. The Canadian Forest Service uses class Gs&Cs to strengthen the forestry sector. For example, a class contribution (CFS) was used to support the development of a roadmap for the Canadian Wood Council. As a result, the Council changed its fee structure to obtain long-term investment for its code work. This, in turn, enabled the Council’s codes and standards work on the use of wood in non-residential applications. This work influenced the Wood First Policy in British Columbia which included changes in the building code to allow wood structures to reach six storeys, compared to the previous four storeys. Another contribution was used to explore possible mechanisms to facilitate the Canadian wood industry to access new export market opportunities. The project created a dialogue among stakeholders about a possible role for the Canada wood industry in disaster relief response.Footnote 38

The Major Projects Management Office (MPMO) used a class contribution to build the capacity of an aboriginal organization. The funding resulted in the development of a template to be used by aboriginal groups when negotiating with industry on mining agreements. This supports NRCan’s priorities for competitive resource sectors and sustainable resource development in the North, priorities that are aligned with NRCan’s strategic outcome – economic competitiveness.Footnote 39

The evaluation reviewed a number of projects that contribute to the strategic outcome, environmental responsibility.Footnote 40 For example, a class contribution supported a CanmetENERGY multi-year study that collected energy and comfort data from zoned and non-zoned homesFootnote 41 and surveyed homeowners for their feedback on whether zoning was effective. The class contribution was used to fund McMaster University’s field tests to collect data from the new and retrofitted homes with Heating Ventilation Air Conditioning (HVAC) zoning. The purpose of the contribution was to assess – through field measurements, utility data, and occupant surveys – various performance indicators relative to “Zone Comfort” technology in occupied homes and in retrofit applications. The data showed that zoned homes used 15 percent less electricity and 7 percent less natural gas than non-zoned homes. The data will be used to develop modelling capabilities to enhance home energy analysis software and is expected to be useful in promoting the benefits and cost-savings of zoned systems to builders and homeowners. The results of this project are also being disseminated through the Association of Heating Manufacturers.

Interviews and case study evidence suggest that if projects had not received funding, either the project likely would have been reduced in scope, delayed or would not have occurred at all. Funding recipients stated that NRCan funding facilitated leveraging of other funding, enhanced the visibility and credibility of their projects, and provided more time to focus on program delivery as opposed to fund-raising activities. Those receiving funding for R&D projects reported that NRCan funding expedited market development and commercialization, and enabled a broader research scope and expansion of testing and development work. R&D proponents also noted that in the current economic climate when venture capital is particularly scarce, government funding has become even more critical in helping small- and medium-sized enterprises to build their R&D capacity.

Funding recipients indicated that some features of the Class Gs&Cs Program also contributed to outcomes achievement:

  • range of eligible costs included in the terms and conditions helps to cover a variety of program costs (e.g., office operating and maintenance expenditures) that may not be provided by other funds, enhancing the value of class funding; and
  • multi-year funding enables support for longer-term projects facilitating outcomes achievement.

3.2.3 Achievement of Intended Outcomes

Evaluation Question Methods Assessment
To what extent has the Program achieved its intended outcomes? Interviews; case studies; file review; database review. Evidence suggests that the Program has contributed to some outcomes, but there is insufficient evidence for achievement of other outcomes.

Summary:

The evidence suggests that the Program has contributed to “increased interest with respect to priority areas or energy efficiency and sustainable development activities”, “increased knowledge in natural resources policy and priority areas or natural resource industry sectors”, and “new or improved technologies”.

While there are examples of achievement related to the outcomes “increased awareness of fields of study and careers”, “policy development”, and “technology transfer”, the extent of achievement is not clear given the available evidence.

Outcomes 1&2: Increased interest with respect to priority areas or energy efficiency and sustainable development activities / Increased knowledge in natural resource policy and priority areas/industry sectors

Without more direct assessments of “increased interest” or “knowledge gain” or comprehensive self-report data from end-users,Footnote 42 these outcomes are difficult to assess. Evidence from the file review (using knowledge output data as proxy indicators), case studies and interview data suggest that the Program contributed to these outcomes through development of knowledge, promotion and dissemination activities.

Case study, interview and file evidence show that the majority of class agreements have resulted in knowledge outputs. An examination of 42 filesFootnote 43 indicates that:

  • ten of the agreements funded conferences or workshops;
  • nine agreements provided data or information to be used for databases, websites, models/simulations, or tools;
  • eight yielded scientific/technical research results;
  • seven agreements produced social/policy research results;
  • five provided support for International Energy Agency implementing agreements and tasks; and
  • four involved technology development.

The file review indicates that the majority of agreements produced the expected outputs and milestones or were on track to produce outputs.Footnote 44 While this does not provide direct evidence of outcome achievement, it indicates that some of the conditions are in place (i.e., knowledge outputs) to support outcome achievement pertaining to increased interest and knowledge.

Interviewees also note that international activities leverage considerable access to global expertise and information. In a review of the 217 class grants, 61 (28 percent) supported international activities. Of these 61 grants, over half (38) were directed to developing and improving access to information (e.g., developing and maintaining international iron ore production and trade statistics; production of international seismological bulletin series; support for the collection, analysis and publishing of standard earthquake information; and support for the International Energy Agency’s Carbon Capture and Storage Roadmap Report). The remaining 23 class grants supported conferences or forums (7); international R&D activities (6); implementing agreements (6) (e.g., Task 19: Hydrogen Safety); and membership fees for international organizations (4).

Interview and case study evidence highlighted a number of publicly available educational materials developed through the support of the Program such as the Canadian Geographic Watershed Module (class contribution funding of $50,000), and an interactive energy map of Canada available on the Energy Portal.Footnote 45 The Pellet Handbook ($130,000) was developed in collaboration with the CANMET Bioenergy Group and the Wood Pellet Association of Canada. The guide covers all aspects of biomass pellet production and utilisation and is the result of collective efforts of the members of International Energy Agency (IEA) Bioenergy Agreement. The project was designed to contribute to an increase in pellet utilization within the energy sector by appropriate distribution of information. Task members of the IEA Bioenergy Agreement commented on the sound technical quality and comprehensiveness of the handbook.Footnote 46

Earth Science Sector projects funded through class have generated and disseminated knowledge concerning Canada’s landmass. For example, a class contribution ($10,000) has supported the Canadian Journal of Remote Sensing which provides a forum for the publication of peer-reviewed scientific research. The case study “Land Management Problems in Hydrocarbon Development Areas” acquired data that will be used to model the response of permafrost areas to land use activities related to northern hydrocarbon development, including adaptation to climate change.

A CFS case study entitled “Exploring Delivery Mechanism for Increased Global Use of Wood” contributed to increased knowledge in a priority area (i.e., new market opportunities for the forestry sector) in relation to opportunities for Canadian pre-fabricated wooden shelters for disaster relief and transitional housing.

Interview and case study evidence indicate that R&D results are being disseminated to their target audiences through conferences, technical workshops, publications, and research consortiums comprised of private and public stakeholders. For example, an R&D project related to Solar Canopy Illumination System reached its primary audience (e.g., building industry) through trade shows and exhibitions. This project also received media attention and was covered on the Knowledge and Weather Networks and Radio-Canada as well as in newspaper articles in the Vancouver Sun.

In an R&D project involving the development of advanced proton membrane water electrolyzer stack (IETS), the company is reaching its target audience by regularly updating its potential customers regarding the modifications and improvements to its technology. In another example, class contributions (ES) were used to support training in remote areas which increased reachFootnote 47 to the Heating Ventilation Air Condition (HVAC) industry and improved knowledge in the design and installation of energy-efficient HVAC systems.

Outcome 3: Increased awareness of fields of study and careers in the Canadian natural resource industry sectors

A review of the class grants and contributions database (N=732) indicates that a small percentage (8 percent) of contributions and 22 percent of grants are directed to supporting students and highly qualified personnel in fields of study and careers in natural resource sectors.Footnote 48 These agreements include support for scholarship programs, awards and competitions, and support for the research of graduate students in natural resource fields of study.

Two case studies were relevant to this outcome. A class contribution entitled “Land Management Problems in Hydrocarbon Development Areas” contributed to the Masters thesis of a McGill University graduate student involved in the R&D work. From the university perspective, this type of project also helps attract future graduate students to its geology program as it provides students with opportunities to work in both science and technology. The second example shows that class contribution funding indirectly supported this outcome by strengthening the relationship between a company (dPoint Technologies) and the University of British Columbia. The company sponsored students during the course of the research and hired some of these students.

Outcome 4: Policy development

The Class Gs&Cs Program generates or disseminates knowledge and data that can be used to inform policy development. Interviewees noted that class Gs&Cs funding facilitates access to information important to policy development. For example, “Building Paths to a Low-Carbon Society” provided a forum for industry, scientists, academics, and provincial and federal policymakers to discuss energy policy issues; and the Sustainable Forest Management Network provided the CFS access to current scientific information. The knowledge that was gained through the R&D study on “Productivity, Water Use and Sustainability of Hybrid Poplar Plantations”Footnote 49 is expected to provide relevant information to policy makers as it has implications for climate change policy. The carbon sequestration data from the project is being used by some stakeholders such as Manitoba which has a five-year hybrid poplar plantation program.

Interviewees note that support for international activities, facilitated through class Gs&Cs, provides access to global expertise and information to further NRCan's policy objectives with other countries. Moreover, they provide opportunities to influence deliberations which can influence Canadian industries.Footnote 50 Support for fora such as the World Energy Congress provides opportunities for senior management to participate and to meet bilaterally with key stakeholders. Such meetings, according to interviewees, can result in positive outcomes and, in one example, facilitated a memorandum of understanding with another country.

Outcome 5: Technology development

A review of 227 class contributionsFootnote 51 supporting R&D activities shows the distribution of projects by research type:

  • 37 percent were applied research (n=84);
  • 31 percent of projects were focussed on technology development (n=71);
  • 20 percent were earlier-stage research (n=46);
  • 9 percent were policy research (n=21); and
  • 2 percent involved support for international R&D (n=5).

The evidence suggests that the Program is facilitating technology development. Interview, case study and file evidence provided several examples of R&D projects that contributed to improved technologies or methodologies. As noted above, a considerable number of R&D projects (71) were focussed on technology development.

Case studies highlighted a number of examples of technology development. The Solar Canopy Illumination System (IETS) integrates sunlight with electric lighting. It is a core sunlight smart system that provides sunlight to interior regions of multi-floor office buildings in order to reduce the need for electric lighting and therefore reduce greenhouse gas emissions. Refinements in the technology include the use of lenses instead of mirrors which reduces the size of the canopy making it easier to install. Measures of energy usage followingthe installation of a demonstration project also showed reduced energy consumption. Another project entitled “Land Management Problems in Hydrocarbon Development Areas” (ESS) resulted in improved tools and methods for assessment and detection which will be useful for evaluating environmentally sensitive areas with permafrost.

Outcome 6: Technology Transfer

While technology transfer was an important part of many projects reviewed for this evaluation, the extent that the Program contributed to technology transfer is not clear given the available evidence.Footnote 52 There are, however, a number of successful examples derived from interview and case study evidence:

  • demonstration of locally sourced biomass replacing coal in cement industry ($100,000) contributed to understanding of technical difficulties associated with use of biomass and its impact on cement (IETS);
  • knowledge gained from Hartley Bay project ($137,000) contributed to the advancement of metering infrastructure in two remote communities (IETS); and
  • the Home Builder’s Association “Local Energy Efficiency Partnership” (LEEP) and TAP project ($160,000) involved a technology scan and selection of the most promising energy efficient technologies for housing; this information contributed to the demonstration of 40 energy efficient houses (ES).

Improvements in the reliability and durability of an electrolyser stack – supported through the class contributionFootnote 53 – enabled the Hydrogenics Corporation to demonstrate a scaled up Advanced Proton Exchange Membrane Water Electrolyzer (PEMWE) stack.Footnote 54

Another project illustrated good progress towards improving performance targets in terms of reduced size and costs of fuel cell humidifiers and commercialization.Footnote 55 In addition, two of the improved technologies (portable power humidifier and a 1kW stationary power humidifier) have progressed to commercialization. The company is selling thousands of portable power humidifiers, and the 1kW stationary power humidifier is in low-volume production.

3.2.4 Unintended outcomes

Evaluation Question Methods Assessment
Has the Program resulted in any unintended outcomes? Interviews; case studies. Evidence suggests that there was dependence of some organizations on class funding, but mitigation strategies were employed.

Both Energy Sector and Earth Sciences Sector interviewees noted that, in some cases, there appeared to be a dependence on NRCan funding provided through the Class Gs&Cs Program. Consequently, funding provided to organizations was re-assessed in 2010-11. As a result, the Earth Sciences Sector ceased funding to some organizations. The Energy Sector now requires organizations to submit sustainability plans.

3.3 Performance (Economy and Efficiency)

Evaluation Question Methods Assessment
To what extent has the Program demonstrated efficiency and economy? Interviews; case studies; file review; document review. There are indications that the Program is efficient and economic, but that there is room for improvement.

Summary:

There are indications that the Program is efficient and economic, but that there is room for improvement. The Class Gs&Cs Program reduces the need to create numerous terms and conditions. In addition, interview and case study evidence suggest that contribution outputs were completed in a timely manner. There was evidence that the Program leveraged other funds and in-kind contributions.

Suggestions to improve the Program related to the strengthening of accountability and performance measurement, streamlining of the approvals process, consideration of a retroactivity clause, and improved communications and training.

The evaluation found aspects of the Program’s design and delivery to be efficient and economic:

  • reduced need for creating numerous terms and conditions for funding requests; this also facilitates timely ability to respond to new opportunities;
  • many interviewees also noted that multiple-year funding, afforded through the Program, contributed to efficiencies;
  • interview and case study evidence suggests that outputs were completed in a timely manner; and
  • the Program leverages funds and in-kind contributions from other sources; most of the agreements reviewed indicated some leveraging of funds or in-kind contributions (34 of 42), although the terms and conditions do not require matching of funds from other sources.

An analysis of the files (n=42)Footnote 56 suggests that the Program leverages other funding. Grant and contribution agreements in the reviewed files indicate that for the $6.5 million of class funding spent, $21.2 million of funding was leveraged from other sources (i.e., industry, universities, other federal government departments, provincial governments and international organizations) representing a leveraging ratio of 1:3.Footnote 57 Because of the nature of the stratified sampling processFootnote 58, this leveraging ratio may be exaggerated in favour of funding from other sources.

Program representatives noted a downside to using class Gs&Cs for programs such as Geo-mapping for Energy and Minerals because they have different renewal cycles, creating uncertainty around program renewal decisions. In addition, there is the potential for performance monitoring and evaluation to overlap when there are separate performance measurement requirements for the program and the class Gs&Cs.

Interviewees considered the administrative requirements of the class Gs&Cs to be the same or similar compared to other grant and contribution programs. For example, the Class Gs&Cs Program has the same requirements for sign-off as compared to other G&C programs in accordance with the Departmental Delegation of Financial Signing Authorities document.

There were mixed views expressed as to the efficiency of the approvals process for smaller agreements, likely resulting from different approvals processes across sectors or branches. Some interviewees noted that the approvals process is too burdensome for lower-risk projects and, consequently, they avoided using the Program for smaller funding amounts. A few interviewees also reported that the approvals process for agreements over $100,000 was lengthy, with COE review and approval sometimes taking between one and four months. It was also noted that payment delays can create an undue burden on recipients.

Some of the delays in approvals were attributed to inexperience and lack of knowledge regarding the class Gs&Cs process and contribution agreement template. Interviewees in those sectors that rarely use class, or that do not have access to administrative G&C support within their sector, indicated that they spent a lot of time learning about the process and preparing agreements. Delays in approvals for larger agreements were also attributed to frequent staff turnover at the COE.

Energy and Innovation and Energy Technology Sector interviewees indicate that timeliness in finalizing class grants and contributions is also affected by delays in allocating budgets within their sectors.

Accountability and performance measurement

Many interviewees were uncertain as to roles and responsibilities for the Program (e.g., the responsibilities of the COE vs. the sectors). Most interviewees considered accountability at the agreement level to be appropriate (e.g., sign-off procedures were reported to be followed) but suggested that, overall, program accountability needs to be strengthened through improved oversight and clear roles and responsibilities. For example, responsibilities and process for decisions as to the use of class in lieu of a unique G&C program authority were unclear.

Interviewees also noted that efforts to strengthen accountability should recognize the need to balance flexibility with accountability and indicated that a shift towards centralization of the Program would create an undue administrative burden.

With respect to performance monitoring, NRCan maintains a grants and contributions database to track financial information. In accordance with the performance measurement strategy, performance information contained in progress and final reports for contributions is output, rather than outcome focussed. However, the quality of the output data contained in project reports was not consistent. For some projects, particularly one-off projects involving small amounts of funding, output information is likely all that can be reasonably expected. Given that larger projects may yield some outcome information, at least at the immediate level (e.g., increased knowledge, increased awareness, new or improved technologies), the performance measurement strategy could be revised to ensure appropriate monitoring of outcomes for higher risk and/or larger contributions.

When the Class Gs&Cs Program is used to support other programs such as PERD or GEM, the projects are integrated with the program’s accountability and performance measurement structures. The Office of Energy Efficiency and the Canadian Wood Fibre Centre link class project results to their existing performance measurement systems.

Performance and financial information for the Class Gs&Cs Program was not aggregated and reported to senior management until 2011 as a result of the implementation of an improved management / accountability structure.

In terms of enhancing performance measurement, some interviews suggest that project managers could complete a simple performance report / template for the project. This output and outcome information could be aggregated and reported at the sector level and centrally to the Transfer Payments Review Committee.

Recent changes to management / accountability

The Corporate Management and Services Sector has implemented the following changes in response to the audit recommendations:

  • a management framework was approved by the TPRC in July 2011 which outlines roles and responsibilities of the sectors;
  • the Corporate Management and Services Sector is reporting financial information to the TPRC on a semi-annual basis; in addition, as part of the COE monitoring exercise, a sample of class Gs&Cs will be reviewed and reported to the TPRC on an annual basis;
  • new Treasury Board submissions are sent to the TPRC for review; and
  • tools, guidelines and training for class have been updated; revised tools are available on the Intranet.

It should be noted that the evaluation findings do not reflect the impact of the accountability and management changes made in response to the 2011 audit recommendation, as these changes occurred after the evaluation timeframe of March 31, 2011. However, the evaluation recommendations take into account the changes and consider how they can be further strengthened.

Suggestions to improve Program

In addition to suggestions for strengthening accountability and performance measurement; interviewees also offered the following suggestions:

  • streamline the approvals process;
  • improve communications and training; and
  • include retroactivity clause in the class terms and conditions.

Approvals Process

Some interviewees suggested that the approvals process could be streamlined for smaller projects ($25,000 and below) by allowing a lower level of authority for approval. Suggestions for streamlining the approvals process for larger agreements included improved training for project managers preparing agreements and ensuring timely review by COE.

Communications and Training

A number of interviewees noted that communications between the Centre of Expertise and the sectors needs to be enhanced to ensure clear and timely notification regarding changes to process, rules, templates, tools and reporting for class Gs&Cs.

Interviewees indicated that there was a need to increase awareness of the Program, particularly for those with less grant and contribution experience, and to provide concise, plain-language guidance and tools with respect to:

  • the rationale and objectives of class grants and contributions (in comparison to other Gs&Cs authorities);
  • description and overview of class terms and conditions;
  • preparing contribution agreements for class;
  • clear guidance on process – who to contact, what forms to use, approval steps;
  • definition, examples and mitigation strategies for potential “conflict of interest”;
  • recipient audits – who arranges for and conducts these recipient audits; and
  • performance measurement concepts and simple reporting templates for project managers.

Retroactivity

NRCan interviewees suggest that, commensurate with the risk of the project, payments retroactive to the approval of the contributionFootnote 59 should be permitted. Interviewees note that many of the budgeting and approval delays are difficult to address and that a retroactivity clause would provide the most feasible solution. It was indicated that risks associated with a retroactivity clause could be mitigated by providing appropriate controls (e.g., the applicant indicates willingness to assume the risk and is advised in writing of the risk being assumed). In summary, interviewees would like the class grants and contributions authority to allow retroactive costs with the risk assumed by the proponent.

Best Practices and Lessons Learned

Evaluation Question Methods Assessment
Are there any best practices and lessons learned from the Class Gs&Cs Program? Interviews; case studies. Some good practices were demonstrated.

Summary:

There were some noted good practices derived from case study and interview evidence. These include clear and regular two-way communication between the project manager and recipient; using smaller agreements to test ideas; recognition of expertise and skills of project manager in facilitating outcomes achievement; and sharing results of recipient audits among managers and staff.

Interview and case study evidence highlighted some good practices and lessons learned with respect to management, accountability, and performance measurement.

With respect to facilitating successful outcomes from contribution agreements, project managers and project recipients underscore the importance of the following:

  • clear and regular two-way communication between the project manager and the recipient;
  • that smaller agreements can be used to test ideas; and
  • in addition to funding, the expertise, knowledge and skills of NRCan project managers are contributing factors to success.

In terms of managing class grants and contributions, branches and sectors have found it useful to have a designated grants and contributions administrator to support the processing of agreements.

The Office of Energy Efficiency also has designated a representative to provide additional oversight of class Gs&Cs over $50,000. The Branch also posts results from recipient audits for contributions on the Intranet.

4.0 Conclusion

In conclusion, the Class Grants and Contributions Program is relevant and supports NRCan’s strategic outcomes, particularly with respect to economic competitiveness and environmental responsibility. Its broad scope facilitates access across the Department to grants and contributions, allowing managers to respond to emerging opportunities and priorities. Its use has also supported existing programs when there is no other appropriate G&C authority.

However, the broad nature of the Program also creates challenges in terms of accountability and performance measurement. The Centre of Expertise implemented a number of changes in 2011 to improve management and accountability. The evaluation found that the relevance and performance of the Program could be enhanced by further strengthening these changes to ensure clarity of roles and responsibilities, improved performance measurement, monitoring and reporting, and continued implementation of a communications and training strategy.

Annex A: Class Contribution Case Studies

 
Case Study Title and Description Reason for Selection
Exploring Delivery Mechanisms for Increased Global Use of Wood (CFS Project)

Purpose: to explore possible delivery mechanisms to increase the global use of Canadian pre-fabricated wooden shelters in disaster reconstruction and international development.

NRCan: $131, 330:
CMHC: $50,000 in-kind; $25,000
One-year (2010-2011).

Other partners and stakeholders:
FPInnovations, Manufactured Housing Institute of Canada (MHI), Shelter Canada,
Canadian Mortgage and Housing Corporation
Recommended by program representatives as positive example contributing to knowledge of opportunities for forest sector.
Productivity, Water Use and Sustainability of Hybrid Poplar (HP) Plantations in Western Canada (CFS Project)

Purpose: to better understand the factors affecting the growth, water use, and carbon sequestration potential of hybrid poplar plantations in Western Canada.

NRCan: $93,475;
UBC: $24,000;
One-year (2010-2011). Proponent: University of British Columbia
Recommended by program representatives as positive example contributing to new knowledge, technology development.
Canadian Interagency Forest Fire Centre (CIFFC) (CFS project)

Purpose: towards cost-sharing of on-going operational forest fire management services by CIFFC.

NRCan: $205,000;
CIFFC & provinces / territories: $467,784;
One-year 2008-09.

Note that ongoing annual support provided through class contribution funding. Support provided based on agreement with provinces and CIFFC since 1984.
Proponent: CIFFC
Selected by evaluation team using sector, materiality, project type (promotion), type of recipient (NGO).
Canadian Centre for Energy Information (ES project)

Purpose: to enhance its web-based energy information portal in order to provide and maintain high-quality, factual, relevant and timely information on Canadian energy and energy-related issues

NRCan: $600,000 (budgeted);
Proponent: $3.2 million;
Multi-year: 2008-09 to 2011-12.
Selected by evaluation team based on sector, materiality, project type (promotion).
Heating, Refrigeration and Air Conditioning Institute of Canada (HRAI) (ES project)

Purpose: to support and promote the proper design and installation of residential mechanical systems in Canadian housing by providing subsidies.

NRCan: $310,500;
Multi-year: 2008-09 to 2010-11.
Recommended by program representatives as positive example contributing to energy efficiency objectives.
Land Management Problems in Hydrocarbon Development Areas (i.e. areas rich permafrost) (ESS project)

Purpose: To improve detection and assessment of land management problems in hydrocarbon development areas with ice-rich permafrost.

NRCan: $92,800;
Proponent (McGill Univeristy): $97,000 in-kind;
Multi-year (2009-10 to 2010-11).
Selected by evaluation team based on sector, materiality, and project type (R&D) criteria.
Development of Advanced Proton Exchange Membrane (PEM) Water Electrolyzer Stack and System for Hydrogen (IETS project)

Purpose: to improve the cost-competitiveness, reliability and durability of a novel proton exchange membrane water electrolyzer (PEMWE).

NRCan $725,000;
Multi-year: 2008-09 to 2010-11.
Suggested by program representatives as a positive example of technology development and transfer.
Solar Canopy Illumination System (IETS project)

Purpose: to enhance the current design of a system that provides sunlight to interior regions of a multi-floor office building, install it in six buildings, and monitor its performance.

NRCan $75,000;
Partners/proponent: $4.9 million (research consortium of public and private stakeholders);
Multi-year: 2009-10 to 2011-12.
Selected by evaluation team based on sector and project type (R&D).
Development of Fuel Cell Humidifiers and Commercialization (IETS project)

Purpose: To develop beta and pre-production gas-to-gas pleated humidifiers for use in automotive and stationary power applications that meet required performance targets, reliability, size and cost requirements.

NRCan:$730,000;
Multi-year: 2008-09 to 2011-12.
Suggested by program representatives as a positive example of an R&D project that has progressed towards commercialization.