Evaluation Report: Targeted Geoscience Initiative 4 (TGI-4)

Table of Contents

Abbreviations and Acronyms

Acronym Meaning
CAMIRO Canadian Mining Industry Research Organization
CCCESD Council of Chairs of Canadian Earth Science Departments
CMIC - EIC Canada Mining Innovation Council - Exploration Innovation Consortium
GAC Geological Association of Canada
GSC Geological Survey of Canada
HQP Highly Qualified Personnel (Professional)
IGA Intergovernmental Geoscience Accord
MAC Mining Association of Canada
MAC Mineralogical Association of Canada
NGSC National Geological Surveys Committee
NRCan Natural Resources Canada
PDAC Prospectors and Developers Association of Canada
PTGS Provincial and Territorial Geological Surveys
TGI-4 Targeted Geoscience Initiative Phase 4

Acknowledgements

The Evaluation Team would like to thank those individuals who contributed to the TGI-4 evaluation, particularly members of the Evaluation Advisory Committee, Earth Sciences Sector management, as well as other contributors who provided insights and comments crucial to this evaluation.

The TGI-4 Evaluation Project Team was managed by Jamie Riddell, with Michelle Brazil and Jarret Boon of the Strategic Evaluation Division providing evaluation support. Jennifer Hollington, Head of Evaluation at NRCan and Gavin Lemieux, Director, provided Senior Management oversight. TDV Global Inc. provided evaluation services for the project.

Glossary

The TGI-4 Program touches on geological and technological scientific topics. The following glossary was compiled to help the reader with some terms. Additional information has been footnoted throughout the report.

 

Executive Summary

This report presents the results of the evaluation of the Targeted Geoscience Initiative 4 (TGI-4) undertaken by Natural Resources Canada (NRCan) between August 2013 and June 2014. TGI-4 is located on the NRCan Program Alignment Architecture (PAA) as Sub-Program 1.3.3.

TGI-4 is a 5-year research program focusing on deep exploration ending on March 31, 2015. It is funded by C-base and A-base resources totalling $47.4 million. The evaluation covered the period from 2010-11 to 2013-14, representing a program total of approximately $21.2 million in C-base expenditures and $18.3 million in A-base expenditures. The evaluation examined core issues related to relevance and performance as per the Treasury Board (TB) Policy on Evaluation (2009).

New mineral resources, particularly in mature mining districts, are likely to be found deeper below the earth’s surface, which makes them difficult to detect using traditional methods. The objective of TGI-4 is to promote the discovery of new ore reserves by providing state-of-the-art public geoscience knowledge and techniques to focus on deep exploration for buried deposits in and around established and emerging mineral districts. The Program is delivered through eight thematically based projects that bring together industry, academia and federal, provincial and territorial governments in a collaborative research approach to produce new tools, techniques and models for deep mineral exploration.

Mineral exploration and natural resource development is a shared responsibility among various stakeholders, including the federal government, and provincial and territorial governments. The roles of government organizations in mineral exploration have been clearly articulated in the Intergovernmental Geoscience Accord (IGA). Further, the Program benefits from the participation of industry and academia in TGI-4 research activities.

The evaluation findings presented in this report are based on evidence collected from a variety of sources. A document review, an international comparison and literature review, an online survey, two case studies, and 36 in-depth interviews (with internal and external stakeholders) were conducted. This multiple-lines-of-evidence approach allowed for corroboration of data and increased reliability of the findings.

Relevance
In summary, there is a well-defined need and role for the federal government in the production of public geoscience, and TGI-4 is well aligned to government and NRCan objectives and priorities. NRCan has both the mandate and expertise to deliver programs to address the need for public geoscience. TGI-4’s collaborative approach helps ensure industry needs are being addressed through relevant research that is well coordinated with other initiatives.

Need for the Program
The Program addresses a continuing need for public geoscience with respect to deep mineral exploration in Canada, given that knowledge gaps still exist and new research questions are being identified. TGI-4 program outputs are aligned to industry's needs for public geoscience related to deep mineral exploration. Survey evidence supported the finding that the Program was meeting industry needs: 90% of industry respondents in the stakeholder survey indicated that TGI-4 was addressing their organization's needs. The alignment is facilitated by the collaborative approach undertaken by TGI-4.

Alignment with Government Priorities and NRCan Strategic Objectives
TGI-4 is consistent with the government priorities related to economic growth, as outlined in the Speech from the Throne 2010 and Budget 2010. It is aligned to Strategic Objective 1 of NRCan's Program Alignment Architecture (PAA), "Canada's natural resource sectors are globally competitive" and is linked to the NRCan priority of "unlocking resource potential through responsible development."

Alignment with Federal Roles and Responsibilities
TGI-4 aligns with the federal government’s role and legislated mandate in the provision of public geoscience. TGI-4 is consistent with the role of the federal government as defined by the IGA, which is "thematically based and national in scope and significance."

Performance
In summary, the Program has produced a wide range of planned outputs, and has identified others that are on track to be produced. The Program Theory as expressed in the Logic Model was designed to reflect intermediate and ultimate outcomes that extend beyond the five-year funding to the end of 2015. However, there is strong evidence that the Program is making progress in achieving two of its three immediate outcomes within the five-year period. There are indications of progress towards achieving one of the intermediate outcomes as of March 2014, given that there is evidence of some early uptake by industry of TGI-4 outputs that have influenced exploration activities and strategies.

The production and effective dissemination of final synthesis research products will be critical to achieving the expected results of the Program. Information dissemination processes should be strengthened to maximize the impact of these products. In addition, given the increasingly prominent role of academia in program delivery, it is important to address outstanding issues regarding consultation and administrative arrangements between NRCan and academic institutions.

Achievement of Expected Outcomes
The Program is producing expected outputs, but at the time of the evaluation, many are preliminary in nature and final products are planned for the last year of the Program (i.e., March 2015) or later. The Program encountered initial delays that impeded project implementation and accounted in part for delays in the production of outputs. The result is that the full body of knowledge expected from the Program, while on track, has not yet been fully produced or disseminated. Dissemination activities will have a significant impact on the achievement of expected outcomes, particularly the intermediate outcomes.

There was evidence that the Program is making progress in achieving its immediate outcomes. Understanding of distal indicators to ore environments increased due to TGI-4 research. New technologies and methods have been developed and there is evidence of uptake by companies involved with the research activities. The immediate outcome, "network of Canadian research centres," became redundant with the launch of the industry-led Footprints project, making direct attribution of achievement to TGI-4 less clear. It is very clear, however, that the TGI-4 program resulted in increased collaboration and networking between industry, academia, and government scientists.

At this stage in the Program, the evidence suggests that it is likely the Program will achieve its expected intermediate outcomes, provided its dissemination strategy is improved. The research projects are on-track, and all stakeholders expressed the expectation that the final research results would be beneficial, based on the information available thus far. By directing attention to the dissemination of information in particular, the Program will be in a better position to provide evidence to more fully assess the intermediate outcomes.

Demonstration of Economy and Efficiency
TGI-4 is a well-managed program using sound, science-based project management techniques and tools. Financial and performance information is collected and monitored on a regular basis, although there is still a considerable amount of performance data to be collected in 2014-15 according to the Program's performance measurement plan.

Overall program management costs are reasonable, representing approximately 10% of the total program budget. The Program’s design as a multi-stakeholder and collaborative effort enables the leveraging of resources and in-kind contributions. It also presents a good foundation on which to base improvements in information dissemination and stakeholder engagement.

The slow program start-up has delayed the production of some final products until March 2015.

Recommendations and Management Response

Recommendations Management Response Responsible Official/Sector
(Target Date)

Recommendation 1: Ensure the completion and delivery of high-quality, final products that synthesize the project results.

The production of outputs by the Program has been significant to date, but many of the outputs are preliminary or interim in nature. To maximize the results of years of research, it is critical that adequate time and resources be provided to produce and effectively disseminate final products.

Accepted.

As per current practice, TGI has identified and allocated the resources required to complete key program deliverables by the end of the program cycle. In addition, if needed, resources will be allocated beyond the last year of the TGI program to support finalization of additional or complementary scientific publication.

Action: A work plan has been developed and implemented to complete and disseminate the key TGI-4 products by the end of the program in March 2015.

At the start of a new mineral program cycle, ESS will develop a knowledge production plan for all projects informed by systematic understanding of user needs. The plan will identify key products, timelines, and resource allocation. Progress in the implementation of the knowledge production plan will be monitored and reported as part of ESS management reviews.

ADM, ESS

(1) Work plan for the delivery and dissemination of TGI-4 products implemented and key products completed by March 31, 2015.

2) Knowledge production plan for the potential next phase of TGI in place by October 31, 2015.

Recommendation 2: Develop an information dissemination and communication strategy and plan that include the collection of performance measurement data.

Information dissemination and knowledge transfer are critical activities for the Program to achieve its expected outcomes. The information needs to reach the right people in a format that is accessible and user-friendly. A focused information dissemination and communication strategy and plan should be developed.

Accepted.

TGI-4 has released over 300 publicly available geoscience publications and has delivered over 450 scientific presentations at conferences, workshops and events. TGI-4 uses GEOSCAN, the bibliographic reference for the Earth Sciences Sector, which has been updated to provide automatic feed to notify subscribers of new TGI-4 products. The recently renewed TGI-4 Program web page provides direct links to ore system referenced publications. Finally, a TGI-4 compendium CD was developed to offer a direct link to an always up-to-date listing of TGI-4 publications. Nevertheless, a lack of coordination of information dissemination activities at the Program level has impacted knowledge transfer and industry uptake. Over the past 4 years, NRCan dissemination mechanisms, standards and practices were modified to meet the requirements of Open Government policies, which may have created stakeholder confusion as to how to access TGI products.

Clear and consistent communication and planned proactive dissemination both internally and externally are critical to the success of the program and the demonstration that it is contributing to the achievement of expected outcomes.

 

A communication strategy with clear objectives and strategies that are specific for key stakeholder groups ensures effective knowledge transfer and uptake.

Action. At the start of a new mineral program cycle, ESS will develop a communication and dissemination strategy in support of mineral exploration. The strategy will include communications objectives, public environment analysis, strategic considerations, target audiences, communication approach, key messages, products and activities. Progress in the implementation of communication and dissemination strategy will be monitored and reported as part of ESS management reviews.

ADM, ESS

(To be implemented no later than October 31, 2015)

Recommendation 3: Develop a clearly articulated stakeholder engagement strategy.

The Program's multi-stakeholder, collaborative approach is paramount for this type of initiative. Yet TGI-4 experienced some delays while appropriate consultative mechanisms were put in place, for example with the provincial and territorial geological surveys (PTGSs). At the same time, industry is a major stakeholder and academia is taking on an increasingly prominent role in program delivery. To maximize the benefit of consultation, it is therefore recommended that a stakeholder engagement strategy be developed that collects and balances the input of the stakeholder groups at the outset, and throughout, any similar program.

Accepted.

ESS engagement at the program level focuses on existing national organizations that represent these stakeholders groups. Engagement with provinces and territories is conducted under the terms of the Intergovernmental Geoscience Accord (2012) through the National Geological Surveys Committee and the subcommittee on TGI. During and subsequent to the evaluation period, TGI-4 has undertaken enhanced external stakeholder engagement with:

  1.  the mineral exploration industry; through focused workshops (PDAC-NRCan Industry Workshops, Toronto (March 6, 2014) and Vancouver (April 15, 2014), questionnaire (January-March, 2014),
  2. The provincial and territorial geological surveys (PTGS) through bi-lateral (June 2014) and NGSC group meetings,
  3. and academia; presentations and discussion with Council of Chairs of Canadian Earth Science Departments (Ottawa, May 2013).

Action: At the start of a new minerals program cycle, ESS will review the program’s governance structure to include a key stakeholder advisory group as well as regular formal meetings with recognized stakeholder groups (PTGS, industry and academia) to provide input and advice regarding the development and implementation of future program activities related to mineral exploration.

ADM, ESS

(To be implemented no later than October 31, 2015)

Recommendation 4: In any subsequent iteration, develop a Logic Model that is appropriate to the funded period, and continue with special studies to assess long-term outcomes.

The TGI-4 Logic Model identifies outcomes that are to be achieved after the conclusion of the Program. This is appropriate given the longer cycles of research and mineral exploration that are inherent in the sector and this program. However, the logic model needs to clearly link early and intermediate outcomes that are achievable and measurable within the timeframe of the program with long term outcomes that demonstrate the overall impacts of the research and exploration. It is recommended that in any future TGI iterations, realistic outcomes that can be achieved within the timeframe of the funding cycle be set for immediate and intermediate outcomes. In addition, the program should collect cumulative information so that the long-term outcomes can be assessed.

Accepted.

Achievement of ultimate outcomes for programs to support mineral exploration is expected well beyond the end of a program cycle and is dependent on a number of external factors (e.g., economic cycles).
ESS will focus on short and intermediate outcomes in the development of a logic model for future initiatives for program activities related to mineral exploration. The logic model and associated performance indicators will focus on outcomes that are achievable during the program duration and will be developed in consultation with NRCan reporting and evaluation groups.

Action: At the start of the new minerals program cycle, ESS will work with SPI to develop a revised logic model with early and intermediate outcomes designed to be measured and reported on within the timeframe of the program and the link with long-term outcomes clearly established.

ADM, ESS

(To be implemented no later than March 31, 2015)

1.0 Introduction

1.1 Overview

Natural Resources Canada (NRCan's) Targeted Geoscience Initiative 4 (TGI-4) is designed to provide the mineral exploration industry with new ore system models and innovative methodologies that allow more effective targeting of deeply buried mineral deposits. More effective targeting was expected to increase the likelihood of discovery, thereby lowering the risk associated with deep exploration.

The purpose of the evaluation was to assess the relevance and performance of NRCan’s TGI-4 Program, as required by the Treasury Board of Canada’s Policy on Evaluation (2009). This report presents the findings, conclusions and recommendations of the evaluation of the TGI-4 Program over the period 2010-11 to 2013-14.

1.2 Program Rationale

Geoscience knowledge provided as a public good by Canada’s government geological survey organizations is recognized as a key enabler of responsible development of mineral resources, which underpins a significant proportion of the economyFootnote 20 and sustains over a 100 communities.Footnote 21 Canada is experiencing a decline in base metal reserves that has been caused, in part, by the depletion of known shallow deposits and declining rates of discovery of new deposits. 

New mineral resources, particularly in mature mining districts, are likely to be found deeper below the earth’s surface, which makes them difficult to detect using traditional methods. There is a high level of uncertainty around the presence and location of ore deposits that are deeply buried, and the techniques and technologies for identifying potential deep exploration targets are not highly developed.

1.3 Program Description

The objective of TGI-4 is to promote the discovery of new ore reserves by providing state-of-the-art public geoscience knowledge and techniques to focus on deep exploration for buried deposits in and around established and emerging mineral districts. The TGI-4 program is designed within the framework of the Intergovernmental Geoscience Accord (IGA) Footnote 22 and reflects the role and responsibilities of the various geological survey jurisdictions in Canada. In the long-term, TGI-4 is expected to generate significant and sustainable economic opportunities. (Please see Annex A for the TGI-4 Logic Model.)

The work carried out under TGI-4 is intended to produce a range of geoscience knowledge outputs, centered on development of new ore system models or new methodological approaches for the detection of buried ore deposits.

The research is organized into 7 ore system projects and one cross-cutting methodology development project with research activities occurring across Canada on over 50 sites.

  • Lode Gold ore systems;
  • Nickel, Copper, Platinum Group Elements, Chromite systems (Ni-Cu-PGE-Cr);
  • Intrusion-related systems (e.g., porphyry-style deposits);
  • Sedimentary Exhalative ore systems (SEDEX);
  • Volcanic Massive Sulphide systems (VMS);
  • Uranium systems;
  • Specialty metal systems (e.g., rare earth elements); and Methodology Development (geophysics, geochronology and analytical geochemistry).

Figure 1: TGI-4 Project Sites

Figure 1: TGI-4 Project Sites

 
Text version

Figure 1: TGI-4 Project Sites

A simplified bedrock geologic map and a shaded relief map are combined to give a textured geologic perspective of Canada. The features are shown in pastel colours as a backdrop to the Targeted Geoscience Initiative 4 research sites. These sites are distributed across the country in most of the provinces and territories. There are 6 in the North-West Territories, 2 in Nunavut, 13 in British Columbia, 2 in Saskatchewan, 2 in Manitoba, 9 in Ontario, 10 in Quebec, 10 in Newfoundland and Labrador, 4 in New Brunswick, and 2 in Nova Scotia. There are 5 Sedimentary Exhalative, 13 Specialty Metals, 12 Nickel, Copper and Platinum Group Elements, 13 Intrusion related 5 Volcanic Massive Sulfides, 3 Uranium and 9 Lode Gold.

Source: NRCan website

 

Source: NRCan website

1.3.1 Governance, Roles and Responsibilities

The Department of Natural Resources Act, S.C, 1994 and the Resources and Technical Surveys Act, R.S.C. 1985 outline the authority and accountability of the TGI-4 Program. The Assistant Deputy Minister (ADM) and the Earth Sciences Sector (ESS) is accountable for TGI-4. The program operates within the Central and Northern Canada Branch of the Geological Survey of Canada (GSC), and the Central Canada Director of the GSC is responsible for the program. On an operational level, the program is managed by a Program Manager who reports to the Director, Central Canada, GSC. Figure 2 illustrates the organizational chart for TGI-4:

Figure 2: TGI-4 Program Governance

Figure 2: TGI-4 Program Governance

 
Text version

Figure 2: TGI-4 Program Governance

Figure 2 is an organizational chart showing the management structure for TGI-4.  The top level is the Assistant Deputy Minister (ADM), Earth Sciences Sector (ESS). Below the ADM-ESS is the Director General (DG), Central and Northern Branch, Geological Survey of Canada (GSC). Immediately below the DG, is the Director of the Central Region for the GSC.  Reporting to the Director is the TGI-4 Program Manager.  Reporting directly to the Program Manager is an Administrative Assistant, and External Engagement and Communications Officer and a Government Coordination and Program Compliance Officer.  The two Officers are grouped together in the Communications, Coordination and Compliance function. Also reporting to the TGI-4 Program Manager are two Sub-Division Heads: one for Precious Metals and Nickel and one for Base and Other Metals.  These two Sub-Division Heads perform the TGI-4 Project Management function. The reporting relationship is distinguished by a dashed line to show that the reporting relationship is solely for the purpose of reporting on TGI-4 projects.

Source: TGI-4 Program Documentation

 

Source: TGI-4 Program Documentation

1.3.2 Resources

Table 1 shows the total resources allocated to the TGI-4 program between 2010-11 and 2014-15 by fiscal year. A-base funding reflects the staff resources which fluctuated according to the project implementation needs. C-base funds end on March 31, 2015.

Table 1: Total Program Resources

TGI-4: A-Base and C-base resources by Fiscal Year
Amount 2010-11 2011-12 2012-13 2013-14 2014-15 Total
A-base (salaries) $3,916,658 $4,171,012 $5,319,770 $4,849,137 $4,682,827 $22,939,404
C-base $3,050,404 $4,902,535 $4,678,409 $5,859,800 $2,676,287 $21,167,435

Sub-total:

$6,967,062

$9,073,547

$9,998,179

$10,708,937

$7,359,114

$44,106,839
EBP and Internal Services (C-base)           $3,247,131
Total           $47,353,970

Source: SAP, GFI and program documentation

Table 2 illustrates the TGI-4 Program total C-base budget and actual expenditures from 2010-11 to 2013-14. In Budget 2010, TGI-4 was announced as a two-year, $12 million program. In March 2011, an additional three years and $13 million were announced for a total of $25 million over five years. More detailed financial information is presented in Section 5 of this report.

Table 2: Financial Summary C-base resources
  Budget Actual
(March 31, 2014)
Remaining
Personnel 2,774,767 1,711,170 1,063,597
Operations and Maintenance 19,870,279 15,505,913 4,364,366
G&C 1,800,000 1,274,063 525,937
EBP 554,954 370,547 184,407
Internal Services   2,305,055Footnote 23 (2,305,055)
  25,000,000 21,166,748 3,833,252

2.0 Evaluation Objectives and Methodology

2.1 Evaluation Scope and Objectives

The Treasury Board Policy on Evaluation (2009) requires that Government of Canada evaluations examine a program’s relevance and performance (effectiveness, efficiency, and economy).

Between September 2013 and May 2014, the NRCan Strategic Evaluation Division undertook an evaluation of TGI-4. The objectives of this evaluation were to:

  • assess the relevance and performance of the TGI-4 program in meeting its objectives; and
  • make recommendations for consideration by NRCan program and senior management.

The evaluation covered TGI-4 activities and expenditures representing approximately $47.4 million over the four year period 2010-11 to 2013-14. It considered the following five evaluation issues: 

Relevance

  1. Is there a continued need for the TGI-4 program? 
    • Assessment of the extent to which the Program continues to address a demonstrable need and is responsive to the needs of Canadians.
  2. Is the TGI-4 program aligned with Government of Canada priorities? 
    • Assessment of the linkages between program objectives and federal government priorities and departmental strategic outcomes.
  3. Is the TGI-4 program aligned with federal roles and responsibilities? 
    • Assessment of the role and responsibilities for the federal government in delivering the TGI-4 program.

Performance: Effectiveness

  1. To what extent has the TGI-4 program achieved the expected outcomes? 
    • Assessment of progress towards immediate, intermediate and ultimate outcomes.

Performance: Efficiency

  1. To what extent has the TGI-4 program managed resources efficiently and economically?
    • Assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes.

Sections 3, 4, and 5 in this report outline the findings for each of these defined evaluation issues and questions. Findings are presented by evaluation question and sub-question for the TGI-4 program.

For each evaluation question, a rating is provided based on a judgment of the evaluation findings. The rating statements and their significance are outlined below in Table 3.

Table 3: Definitions of Standard Rating Statements
Statement Definition
Demonstrated The intended outcomes or goals have been achieved or met.
Progress Demonstrated    Considerable progress has been made to meet the intended outcomes or goals, and achievement is expected based on current plan.
Progress Demonstrated – attention required Some progress has been made to meet the intended outcomes or goals. Management attention is needed to fully achieve outcome.
Not Demonstrated Limited or no progress has been made to meet the intended outcomes or goals as stated or the outcome is no longer applicable (due to changes in the external environment).

2.2 Evaluation Methodologies and Limitations

2.2.1 Methodologies

For this evaluation, data collection consisted of gathering information from the following sources:

  • document review;
  • interviews;
  • online survey of TGI-4 stakeholders (industry, academia, provinces/territories);
  • literature review and international comparison; and
  • case studies (2).

Summary technical reports were developed for each line of evidence.

Document Review
For the purposes of the Document Review, approximately 200 documents were identified by the program. These documents provided a comprehensive record of the Program and included background information, terms of reference, contracts, financial information, internal and external reports and presentations, program updates and other relevant documents. The scientific outputs of the Program were verified but not reviewed comprehensively.Footnote 24

Interviews
A total of 36 interviews were conducted with NRCan staff and program stakeholders, and an additional seven interviews were held for the case studies. A total of five interview guides were used for the evaluation, and a further two for the case studies. Table 4 outlines the categories of interview groupings that were used for analysis.

Table 4: Interviewees
Category Number of Interviewees
NRCan/GSC Senior Management and Staff (NRCan) 19
Provincial and Territorial Geological Surveys (PTGSs) 5
Industry Representatives 5
Academia 7

Online Survey of TGI-4 Stakeholders
The purpose of the stakeholder survey was to reach TGI-4 stakeholders to assess the relevance of the Program to their needs and priorities, as well as its effectiveness in addressing those needs. The 25 survey questions were developed by the evaluation team with input from the Evaluation Advisory Committee. The survey consisted of a rating scale and/or Likert questions with some open questions to collect respondents’ comments.

A list of potential survey respondents and their contact information was compiled by the evaluation team and validated by the TGI-4 program. A total of 85 of the target audience of 264 responded to the survey, representing a response rate of 34.6%. A total of 22 of the initial respondents exited after the first section on background information, leaving 63 respondents (25% response rate) to address the main survey questions - 18 from Academia, 19 from Industry, and 26 from Provinces/Territories.

Literature Review and International Comparison
A literature search was conducted using key search terms developed with the Project Authority. This search was then repeated with the addition of “Australia” and then “Finland” as the two countries identified for the international comparison. The search was conducted at the NRCan libraries, and almost 70 pieces of literature were identified using the GeoRef and Scopus library databases. Further articles and papers were identified and/or provided by interviewees.

Case Studies
A shortlist of case study topics was proposed by the evaluation team. A case study protocol was then developed that included criteria for case study selection. Based on the criteria, the selected case study topics were the Nickel-Copper-PGE-Chrome Ore System (Ni-Cu-PGE-Cr) and the TGI-4 Business Model, which is referred to in this report as the Thematic Ore System Research Approach. The case studies were conducted using interviews (a total of seven) and document review as the lines of evidence.

2.2.2 Limitations and Mitigation Strategies

There were some limitations in the design and methods for this particular evaluation, and mitigation strategies were put in place to ensure that the evaluation findings can be used with confidence to guide planning and decision making.

The principal limitation was that the Program's data collection for a special study (on developing a long-term performance monitoring framework to measure the contribution of public geoscience to exploration success), coincided with data collection for the evaluation. This created some confusion on the part of industry representatives about the purpose of these similar (but different) exercises, which impacted industry participation in interviews for the evaluation and may have impacted industry participation in the survey.

In order to mitigate this limitation, multiple lines of evidence were used to ensure that a wider section of industry input was identified. Two additional industry representatives were also interviewed for the case study. Further, the survey respondent list was expanded to include all industry project contacts noted in the program’s project documentation.

3.0 Findings on the TGI-4 Program Relevance

3.1 Evaluation Issue 1: Continuing Need

There is a continuing need for public geoscience with respect to deep mineral exploration in Canada. Knowledge gaps still exist, and new research questions are being identified. TGI-4 program outputs are aligned to industry's needs for public geoscience related to deep mineral exploration. The alignment is facilitated by the collaborative approach undertaken by TGI-4.

3.1.1 Who are the stakeholders, and what needs of theirs were addressed by TGI-4?

  • The primary beneficiary of the TGI-4 program is the mining and exploration industry. Over 50 companies participated in TGI-4 research projects, and the TGI-4 program also interfaces with industry at workshops, conferences (e.g., PDAC ConventionFootnote 25, GAC and MAC joint Annual General MeetingFootnote 26, provincial open houses, etc.) and through industry-related research organizations, such as Canadian Mining Innovation Council (CMIC). Industry needs addressed by the Program include research to fill knowledge gaps on deep mineral exploration and training opportunities for the development of Highly Qualified Personnel (HQPs) in geosciences , who have experience relevant to industry.
  • Provincial and Territorial Geological Surveys (PTGSs) are partners in the delivery of the Program. PTGSs and NRCan share common objectives in the production of public geoscience and their roles and responsibilities under this shared responsibility are clearly outlined in the Intergovernmental Geoscience Accord (IGA). PTGSs needs are that federal programs such as TGI-4 operate within the defined parameters of the IGA in the pursuit of public geoscience. PTGSs were involved in TGI-4 project development, which assisted in ensuring alignment to the IGA. PTGSs participate directly in some projects and have cost-shared some research activities. The National Geological Surveys Committee (NGSC) created a TGI-4 Sub-committee which is an important communication and coordination platform for the Program.
  • Academia is also a partner in the delivery of the Program. The TGI-4 delivery model includes the participation of academia in research activities. Academic institutions received 26 grants valued at approximately $1.5M to undertake research activities that were integrated in TGI-4 projects. This funding also supported students in completing their academic work, thereby furthering the development of approximately 90 HQPs.
  • In the long-term, the general public can be considered a TGI-4 stakeholder. The Program produces fundamental public geoscience knowledge that improves Canada's international competitiveness in the mining and exploration sector. New geoscience knowledge can improve the effectiveness of exploration investments, thereby attracting exploration investment that in turn can lead to economic opportunities for Canadians.

3.1.2 Are the needs met?

  • TGI-4 projects were designed with input from industry and PTGSs, which provided the opportunity to influence the scope and activities of the projects to better address their needs. The needs of PTGSs were addressed given that TGI-4 adheres to the IGA. There was ongoing communication and coordination through the NGSC TGI-4 Sub-committee, and public geoscience was being produced that was relevant to P/T jurisdictions.
  • The needs of Academia were addressed through research funding and opportunities for students to conduct research activities. The program supported the professional development of approximately 90 HQPs, and 26 research grants, valued at approximately $1.5M, were provided.
  • Regarding industry stakeholders, TGI-4 produced research that aligned with industry needs. While a longer period of time is required to determine the extent to which industry’s needs were met, the research topics were perceived as being relevant to industry issues. TGI-4 research results were promising though preliminary, and there was evidence of early uptake and use of some TGI-4 outputs. Survey evidence also supported the finding that the Program was meeting industry needs: 90% of industry respondents in the stakeholder survey indicated that TGI-4 was addressing their organization's needs, and 60% of industry respondents indicated that TGI-4 was focused on relevant research issues. This corroborates evidence from other lines of evidence (interviews, case studies) that the Program is meeting industry needs.

3.1.3 Are the needs continuing?

  • Almost all stakeholders indicated in the survey and interviews that there was a continuing need for research and dissemination of public geoscience to improve the effectiveness of deep mineral exploration and reduce exploration risks and costs. In addition, the majority of survey respondents (73.5%) either agreed or strongly agreed that new mineral resources in Canada would principally be found at depth, making research into deep mineral exploration relevant to current and future needs. Over the course of the last four years, TGI-4 research had raised further hypotheses that chart a future research agenda on deep mineral exploration.
  • The need for research on deep mineral exploration is recognized in other countries as well. Other countries are actively investing in deep mineral exploration, often in collaborative approaches similar to TGI-4, involving academia, industry and government. Australia is particularly active in deep mineral exploration research.Footnote 27

3.2 Evaluation Issue 2: Alignment with Government Priorities

TGI-4 is consistent with the government priorities related to economic growth, as outlined in the Speech from the Throne 2010 and Budget 2010. It is aligned to Strategic Objective 1 of NRCan's Program Alignment Architecture (PAA), "Canada's natural resource sectors are globally competitive." Further, it is linked to the NRCan priority of "unlocking resource potential through responsible development."

3.2.1  To which federal government priorities is the TGI-4 linked?

  • The TGI-4 program was announced in Budget 2010 at $12 million for two years. It was included in Budget 2010 under "Building on a Strong Economic Foundation." In 2011, NRCan announced an increase to $25 million and an additional three years to 2015 for TGI-4, quoting the government's commitment to economic growth, competitiveness and job creation. Government statements verified that the mineral and exploration industry was a vital part of the Canadian economy.
  • The renewal of the IGA in 2012 also reinforced the importance that the federal government and provinces and territories placed on the work of the geological surveys. This was the fourth renewal of the IGA since its inception in 1999, and it served as a guide to the different levels of government on their roles in regards to the shared responsibility of natural resource development.

3.2.2  To what strategic departmental objectives and priorities are TGI-4 linked?

  • TGI-4 is aligned to the Strategic Objective 1 of NRCan's Program Alignment Architecture (PAA), “Canada’s natural resource sectors are globally competitive." In both the Departmental Performance Report (DPR) 2012-13 and Report on Plans and Priorities (RPP) 2013-14 (see Table 5), TGI-4 was clearly linked to the NRCan priority of "unlocking resource potential through responsible development."
Table 5: TGI-4 and NRCan's Program Alignment Architecture
Level Description
SO 1 Canada's natural resource sectors are globally competitive
Program 1.3 Investment in natural resource sectors Investing in the development of natural resources is costly and risky due to the uncertainties related to the potential economic viability of natural resources. There are many factors to consider when deciding whether or not to develop a natural resource. In some cases, investors and/or companies lack knowledge on and, thus, are unaware of potential opportunities. The objective of this program activity is to encourage natural resource sector investment by either decreasing the risk of development or increasing knowledge on opportunities. This objective is achieved by providing funding and information on the factors that determine the potential economic viability of natural resources.
1.3 Result: Natural resource sectors have increased investment
Sub-Program 1.3.3 TGI-4 Mineral resources are one of the principal economic drivers in many rural and remote Canadian communities, but known reserves are depleting. In order to sustain economic viability in these areas, new geoscience knowledge and techniques are required to help industry more effectively explore for buried, as-yet undiscovered mineral resources in existing and emerging mining areas. The Targeted Geoscience Initiative 4 (TGI-4) is the fourth generation of this program that develops our understanding of entire mineral systems and provides industry innovative ways for deep exploration, thereby maximizing yield. This initiative targets selected mineral districts across Canada to provide the best examples of Canada's major ore systems and to develop optimal predictive models and techniques for deep exploration. In turn, this program supports the natural resource sector's access to viable investment opportunities.
1.3.3 Result: Industry has reduced exploration risks and costs upon application of NRCan knowledge and/or techniques

Source: NRCan Performance Measurement Framework 2012-13

3.3 Evaluation Issue 3: Alignment with Federal Roles and Responsibilities

TGI-4 aligns with the federal government’s role and legislated mandate in the provision of public geoscience. TGI-4 is consistent with the role of the federal government as defined by the IGA, which is "thematically based and national in scope and significance."

3.3.1  Should the Government of Canada be involved in providing contributions to support TGI-4 activities?

  • There is a well-articulated role for the federal government in the provision of public geoscience. Much of the geoscience information that underpins exploration has the economic characteristics of a public good, and the provision of public geoscience to stimulate exploration is a key element of federal, provincial and territorial mining strategies. As Canada competes for exploration investment dollars, the provision of public geoscience information improves its market competitiveness.
  • The role of the federal government in TGI-4 falls within the legislated mandate of NRCan.Footnote 28 Further, the Government of Canada role in providing public geoscience is recognized and clarified in the Intergovernmental Geoscience Accord (IGA).
  • The vast majority of all survey respondents (86%) either Agreed or Strongly Agreed that the Government of Canada's role in TGI-4 was appropriate.

3.3.2 Are there other parties external to the Government of Canada that could distribute the grants and undertake the program activities that are being delivered by TGI-4?

No. The purpose and pre-competitive nature of the research activities undertaken by TGI-4 preclude external parties from carrying out the same work.

  • TGI-4 is a national collaborative program involving industry, academia, provinces and territories that develops publicly available geoscience products. As such, parties external to the Government of Canada with a stake in the program are an integral part of the program delivery.
  • A key partner in the collaborative approach is the CMIC-EIC and consultation between the CMIC-EIC and TGI-4 occurs in the form of feedback on research programs and the presence of the Director for TGI-4 on the CMIC-EIC Footprints project Board of Directors.
  • The CMIC Footprints project is an industry-driven project similar to TGI-4 in that its research agenda focused on issues related to deep exploration. However, it is unsuitable to distribute the grants and undertake the program activities being delivered by TGI-4 given its limited scope and management of intellectual property. It operates with academia and industry, and is supported by federal (NSERC) and industry funding. Federal and provincial and territorial governments are consulted and activities are coordinated. However, companies involved in the consortium are the principal beneficiaries of the information produced.
  • CMIC Footprints and TGI-4 are separate initiatives that have distinct objectives and program delivery models. TGI-4 focused on pre-competitive information and making it publically available. Individual company interests drive the management of intellectual property and selection of research within the Footprints project.
  • There are important differences in scope and purpose between TGI-4 and CMIC Footprints. CMIC Footprints is researching the development of practical applications and approaches to the acquisition and use of data for identifying ore-system footprints, on partner site locations only. It is geographically focused and is both federally and privately funded. In contrast, TGI-4 uses a mineral system approach to research national, thematic perspectives by comparison of multiple archetypical ore deposits throughout Canada. It crosses mineral districts and jurisdictional and geographic boundaries.
  • The GSC is a member of the CMIC-EIC which provides an opportunity to coordinate research activities on similar ore systems. The Lode Gold project demonstrated how the TGI-4 and CMIC-EIC Footprints project collaborated to enable a broader scope of research to be conducted, and the research roles of each organization: TGI-4, CMIC, and Géologie Québec, were defined in order to avoid duplication of effort. The continued collaborative efforts were supported by the CMIC-EIC as well.

3.3.3  Is the TGI-4 Program optimally situated within the Government of Canada structure?

  • The TGI-4 program is optimally situated within the Government of Canada structure. NRCan/GSC has the mandate, the expertise and the experience to deliver the Program.
  • GSCs reputation as an "honest broker" facilitates information exchange among the stakeholders. The credibility that the GSC has earned over the years facilitates its access to the private property of industry for research purposes. In addition, the Program's national scope transcends provincial and territorial boundaries or any one company's property. This places the GSC in a unique position to deliver the TGI-4 program.

4.0 Findings on the TGI-4 Program Effectiveness

4.1 Evaluation Issue 4: Achievement of Expected Outcomes (Effectiveness)

Summary of Assessment
Outcome Level Outcome Assessment
Immediate Network of Canadian research centres focused on relevant exploration research problems. Not Demonstrated
Immediate Improved understanding of distal ore indicators. Progress Demonstrated
Immediate New methods and technologies that allow better detection of buried ore deposits are disseminated to industry. Progress Demonstrated – attention required
Intermediate Improved exploration models are disseminated to industry. Progress demonstrated – attention required
Intermediate Knowledge and technology adopted by industry results in new exploration strategies. Progress Demonstrated
Ultimate Canada's metal reserves increase through discovery of significant buried metal deposits. Not Demonstrated

The Program is producing expected outputs, but many research products are still preliminary in nature and final synthesis products are planned to be delivered at completion of the Program (i.e., March 2015) or later as in the case of peer-reviewed journal publications. The Program encountered initial delays that impeded implementation which accounted for the fact that many outputs are not yet final. Thus, the full body of knowledge expected from the Program had not yet been produced nor disseminated during the evaluation period, which impacted the full achievement of expected outcomes, especially the intermediate outcomes.

The Program is making progress in achieving two of the three immediate outcomes identified in the Logic Model. With respect to "network of Canadian research centres focused on relevant research problems," this outcome has not been demonstrated by TGI-4 efforts. However, collaboration has increased and informal networks have been formed within projects.

As final products remain to be produced, it is difficult to assess the achievement of intermediate and ultimate outcomes given the current state of implementation and the long-term character of the stated outcomes. At this stage, the evidence suggests that it is likely that the Program will achieve the expected intermediate outcomes, as the research projects are on-track, and existing research results are expected to be relevant to organizations. With attention directed towards effective dissemination of information in particular, the Program will be better able to provide evidence to measure its intermediate outcomes.

4.1.1 To what extent has the Program produced expected outputs?

Progress Demonstrated. The types of outputs produced (e.g., data compilations, mineral system and indicator mineral descriptions, HQPs, etc.) were in line with program and project plans.

  • There were over 300 outputs listed by the Program as of March 2014. Many of the outputs were preliminary or interim in nature with final research products expected in the last year of the Program, 2014-15.
  • There was no single authoritative list of outputs, and there were discrepancies between various program information sources such as the website, RSS feed or information provided by project leads. This can create confusion on the part of stakeholders, and the perception that not all research is being released.
  • There were information dissemination activities and products produced as outputs (e.g., presentations, posters, workshops, etc.) but some external stakeholders were unaware of the extent of TGI-4 outputs, or were aware of only those outputs in which they were directly involved.

4.1.2 Immediate Outcome 1: To what extent was a network of Canadian research centres created? 

Not Demonstrated. This outcome is being met through activity that is managed by external sources using external resources and allocation of TGI-4 resources would result in duplication of effort. While progress is not directly attributable to TGI-4 efforts, the resulting collaboration was beneficial to the program, and there was strong evidence that informal networks were created and strengthened among individual researchers.

  • The immediate outcome “network of Canadian research centres” was ill-defined. The logic model does not identify activities and outputs that support this outcome. Program documentation lacked definition of the network of Canadian research centres and showed no resources being applied directly to its achievement. External survey respondents and all groups of interviewees had a variety of interpretations of its meaning. Interviewees in particular noted that research centres indicated a more formal arrangement and pointed out that their research networks were informal and based on individual research interests, though they were strengthened as a result of TGI-4.
  • On a project basis, informal and unstructured research networks were created. Some of these networks were extensive. As an example, the Ni-Cu-PGE-Cr project created an informal research network that included nine universities, four provinces, 20 companies and 15 GSC scientists.
  • The majority of survey respondents (70%) indicated that TGI-4 research was relevant and that their own research networks had been strengthened (70%) and new research opportunities created (67%).
  • The industry-driven CMIC organization received NSERC CRD funding for its Footprints project. In 2012, the Footprints project was viewed as a “first step in establishing a cohesive research network across Canada that will make the Canadian mineral exploration community more globally competitive”Footnote 29 corroborating the fact that the outcome remained unachieved. The launch of the Footprints project meant that, for the TGI-4 program, pursuing this outcome would be duplication of effort.
  • GSC participation on the CMIC-EIC and membership on the Footprints Project Board of Directors demonstrated a strong collaborative approach. However, the minutes of those meetings indicate that TGI-4 participation remained focused on providing status updates on the TGI-4 program, and it served as a means of obtaining industry feedback on project development as identified in the Result-based Management and Accountability Framework.
  • An NRCan Grant under the Class Grants and Contributions program was provided to the CMIC-EIC Footnote 30 to help establish a “management and delivery structure for CMIC-Exploration instigated research.”Footnote 31 These funds provided support for the development of the Footprints project, but are outside of TGI-4 resources. While progress towards a network of research centres can reasonably be attributed to the NRCan grant, this is not attributable to TGI-4. TGI-4 management prudently did not allocate resources towards activity that would duplicate effort.
  • TGI-4 funds allocated via contribution agreement to the CMIC EIC relating to the development of the Footprints project ($40,000) were aimed at increasing the participation of students in the CMIC research to achieve the output of developing Highly Qualified Personnel (HQP).

4.1.3 Immediate Outcome 2: To what extent was the understanding of distal indicators to ore environment improved?

Progress Demonstrated. Progress has been made on improving the understanding of distal indicators, as evidenced by initial research being produced.

  • According to stakeholders, some of the research seems promising, but it was too early to determine the extent to which the outputs would contribute to a better understanding of distal indicators.

Dissemination of the information is very limited, and in some cases only those involved in the research activities are aware of the information.

  • The production and effective dissemination of the final synthesis products will be critical to fully achieve this result. The final products will compile all the interim outputs into one product that is comprehensive and useful for industry.

4.1.4 Immediate Outcome 3: To what extent were new methods and technologies disseminated to industry?

Progress Demonstrated – attention required. There have been new methods and technologies produced, and they have been well documented. However, information dissemination activities varied and were left largely to individual project leaders to incorporate in their specific project plans. Typical information dissemination activities included conferences, workshops, posters, and presentations, but the reach and effectiveness of the dissemination could not be measured as performance measurement data were not collected. As well, there are significant gaps in the information that is disseminated to industry by the Program. The achievement of this outcome has implications for the achievement of longer-term outcomes in the program’s logic model and theory of change.

  • Program outputs are available on the NRCan website and are catalogued in GEOSCANFootnote 32. There is also an RSS feed that can be subscribed to for receiving outputs as they are released. The list of outputs varied between the website, RSS feed and other lists that were provided by project leaders. This created stakeholder confusion as to the extent of the TGI-4 body of knowledge being disseminated. As an example, in the case of the Rare Metals ore system project, there was no listing of outputs on the website. The reason given was that the project was delivered in its entirety by the British Columbia Geological Survey (BCGS), and publication considerations had not been agreed upon. This represented a considerable gap in expected public geoscience information being disseminated by the Program.
  • There has been some early uptake of TGI-4 outputs, but most stakeholders were unaware of TGI-4 outputs beyond those outputs that were produced by their specific project activities. Less than half of survey respondents perceived the Program as achieving "new exploration models/technologies are accessible and have been disseminated."

4.1.5 Immediate Outcome 3: To what extent did these new methods and technologies allow better detection of buried ore deposits?

Progress Demonstrated. There was no evidence that new methods and technologies had led to the detection of previously unknown buried ore deposits. However, industry uptake of some of the innovations developed by TGI-4, incorporation into exploration strategies of research partners, and the correlation with known deposits in the proof of concept phase, strongly suggested that over time (in the context of the mineral exploration cycle) the new methods and technologies would allow the better detection of buried ore deposits.

  • The full body of TGI-4 knowledge to be produced by the Program has not been completed, published nor disseminated. Nonetheless, there was evidence of some early uptake of TGI-4 outputs by industry that has influenced exploration activities. Examples include the uptake of aerial survey information by MacDonald Mines, and the use of TGI-4 outputs by Mustang Minerals which acknowledged the TGI-4 research in its corporate information.
  • More than 45 innovations (e.g., indicators, methods, technologies) have achieved proof-of-concept, that is, have demonstrated their potential to be applied in private sector exploration. Proof-of-concept, however, is often an early stage in the research process, and there often remains much work to be done before it can be applied in the field. Other information produced by the Program (e.g., gravity gradiometer surveys, etc.) has been rapidly adopted by interested companies.

4.1.6 Intermediate and Ultimate Outcomes

  • The TGI-4 Logic Model is partially based on a research continuumFootnote 33 within the context of an exploration cycleFootnote 34. While the research and exploration cycles are often depicted as linear processes, the TGI-4 program engages the cycles at various stages, not necessarily just at the beginning. Nonetheless, given the time required for both the research and the exploration processes, there is a clear and direct impact on the time required to achieve the Program's outcomes. It is not uncommon for the research cycle (e.g., for technology development) to take 5-8 years, and the exploration-to-discovery cycle to take 10-20 years.
  • The Logic Model itself indicates that achievement of intermediate outcomes is not expected until April 2016, after the Program end-date of March 2015. The Logic Model outlines the Program's results chain as outputs (i.e., tools, technologies, and techniques) that will lead to new models, which in turn will influence exploration strategies that in turn will contribute to the discovery of deeply buried ore deposits. It is not uncommon for such a results chain to take up to 20 years to be achieved.
  • It is difficult to assess the achievement of results that realistically will not occur until after the end of the Program. At best, the evaluation can assess the likelihood that outcomes may be achieved. In that respect, there are indications that the Program is making some contributions toward the achievement of outcomes. Nonetheless, many of the outputs produced to date are preliminary, and the full body of knowledge is not yet produced. Considerable time will be required to finalize outputs, disseminate that information in the final year of the program, and for industry to take up that information and apply it in the field.
  • Recognizing the challenges in measuring long-term outcomes, the Program recently commissioned a studyFootnote 35 to investigate the linkage of TGI-4 activities to exploration outcomes. The study highlighted some of the challenges in assessing performance, including the challenge of attributing public geoscience outputs to exploration success when there are a number of other factors that contribute to that success. The study did conclude, however, that there is a high likelihood that TGI-4 will achieve its intermediate outcomes.

Intermediate Outcome: To what extent have improved exploration models been disseminated to industry?

Progress demonstrated – attention required. In this case, “exploration models” were not defined in the program documentation, and no standard definition in the geosciences lexicon was identified. It was clear that TGI-4 was focused on geologic models and new interpretations to be used in exploration, and that projects are on track to produce substantive new information. In spite of some examples that demonstrate industry awareness of models however, all lines of evidence found that it was too early for the program to have disseminated exploration models to industry.

  • There are examples of early interpretations of models being used by the industry partners directly engaged in TGI-4 research. For example, the Lode Gold project revised a geological model that was being used by the Osisko mining company at its Malartic mine property. However, for the most part, project evidence shows that models are not yet complete, and this is substantiated by interview evidence, case studies, and the survey.
  • The models that would be produced were expected in the final synthesis products of the program. The key products are on track to be produced by the end of the Program in March 2015.
  • Full achievement of this outcome would require enhanced dissemination tracking and coordination by the program; however, a robust dissemination strategy and tracking system were not in place.

Intermediate Outcome: To what extent did knowledge, technology and/or models adopted by industry result in new exploration strategies?

Progress Demonstrated. Despite the aforementioned limitations, there was evidence of industry uptake of TGI-4 knowledge and information that had influenced exploration activity.

  • A special studyFootnote 36 commissioned by the Program revealed that of the 19 industry geologists consulted, 14 said that their exploration strategies had been improved as a result of the Program.
  • The evaluation's lines of evidence supported the finding that there had been some early uptake of TGI-4 outputs that influenced exploration activities, but it was not clear what constituted an "exploration strategy". There were several examples of industry adjusting their exploration activities as a result of new knowledge or information produced by the Program.

Long-term Outcome: To what extent did Canada’s metal reserves increase through discovery of significant buried metal deposits?

Not Demonstrated. To date there have been no new deeply buried ore deposits discovered as a result of TGI-4 activities. This is a long-term outcome, and the TGI-4 Logic Model stated that this result was not expected to be achieved until after 2016. The majority of survey respondents (84%) however, indicated that, in their opinion, TGI-4 had at least some potential to contribute to new ore discoveries at depth.

4.1.7 What are the chief internal and external factors influencing achievement of the TGI-4 program objectives?

  • Externally, a number of factors influenced the program delivery of TGI-4. While the impact on production of TGI-4 outputs was unclear, these factors would in all likelihood cause delays in achievement of outcomes. Many of the external factors slowed program start-up activities resulting in delays in implementation.
  • External factors included a staggered start-up due to two funding announcements: one in 2010 announcing a two-year, $12M program and another in 2011 that increased funding to $25M over five years. This impacted on planning consultations with stakeholders and partners. Other factors influencing the start-up of the Program included delayed Treasury Board (TB) approval (announced in Budget 2010 in March 2010, the Program received TB approval in July 2010); federal and provincial/territorial discussions on the renewal of the IGA in 2010-11; and a federal election in 2010.
  • Another critical external factor that impacted significantly on the achievement of outcomes, and would continue for at least a few more years, was the general economic downturn in the mineral and exploration sector. This directly affected the level of exploration investment and consequently, industry uptake and use of TGI-4 knowledge and information in exploration activities.Footnote 37
  • Internal factors also influenced program delivery. The NRCan Terms and Conditions (T&Cs) for Grants and Contributions (G&Cs) were due for renewal in 2011. Advice on the process received by the Program was contradictory, and resulted in consultations with academia on multi-year agreements resulting in one-year grant agreements. After the renewal of the NRCan T&Cs, the contracts were renegotiated to multi-year agreements as originally planned. This caused delays in establishing the research activities associated with the TGI-4 research projects.
  • The delay in transfer of funding under the G&C program was cited by academic stakeholders as a challenge to their activities. This had the potential to negatively influence the relationship with an important delivery partner.

4.1.8 Have there been any unintended (positive or negative) outcomes?

No positive or negative unintended outcomes were identified by the evaluation.

5.0 Findings on the TGI-4 Program Efficiency and Economy

5.1 Evaluation Issue 5: Efficiency and Economy

Overall, TGI-4 is a well-managed program. Financial and performance information is collected and monitored on a regular basis and its multi-stakeholder, collaborative design enables the leveraging of resources and in-kind contributions. It did experience a slow program start-up, which delayed the production of some products.

5.1.1 To what extent did program design and delivery support achievement of its outcomes?

The overall program design supported the achievement of outcomes.

  • The multi-stakeholder and collaborative approach facilitates the necessary leveraging of resources and expertise of the PTGSs, industry, academia and GSC scientists in pursuit of its research activities and objectives. The thematic approach was also well-aligned to the federal role, as defined in the IGA. There were delays in program delivery, which might impact on achievement of outcomes within the five-year program period.
  • The strategic context of the Program was only partially described as program documentation focused on the external environment. There was a lack of information on the initial program design, and how the Program research agenda was developed. It is not clear how, and if, it evolved from previous iterations of TGI. The lack of strategic context for the Program within the GSC made it difficult to assess the relevance of the current focus on deep mineral exploration and its potential longevity as a focus in future program iterations.
  • Initially, the program design (i.e., thematic and pan-Canadian focus of projects) was challenged by some stakeholders. However an effective communication response by the Program resulted in TGI-4 being well-received and viewed as being aligned with industry needs and the federal role.
  • A distinction was made in the evaluation between program design and TGI-4 project design. While there was no evidence identified that stakeholder input impacted program design, there was, however, significant evidence to demonstrate stakeholder input on TGI-4 project design. Some of those consultations started later in the project planning cycle. In 2010, after the program started, there was initial TGI-4 project planning meetings with industry. In late 2010, the program established a GSC internal Science Advisory Committee and, in early 2011, a TGI-4 sub-committee was formed from National Geological Surveys Committee (NGSC) members. Final TGI-4 projects were confirmed around April 2011, one year after program start.
  • External factors, which have already been noted (in Section 4.1.7), contributed to delays in finalizing the TGI-4 projects and starting program implementation. Internal program management decisions also contributed to delays in implementation. Some of those internal decisions included a long consultation process, budget re-allocations across the various TGI-4 projects, and a decision to redesign many of the TGI-4 projects in the fall of 2010 to better align to the thematic approach.

5.1.2 What measures were taken to achieve efficiencies and/or cost savings? 

The multi-stakeholder collaborative approach of TGI-4 produced opportunities for cost-sharing and leveraging resources. This resulted in approximately $13.5M of in-kind contribution being provided by partners (to March 2014).

  • The TGI-4 projects involved collaboration with stakeholders that leveraged various sources of in-kind contribution and funding to support activities and achieve research objectives. Total in-kind contributions from industry, academia and PTGSs were valued at approximately $13.5M to March 2014. As an example, the Ni-Cu-PGE-Cr project had a budget of approximately $3.3M, and $1.9M was leveraged from in-kind contributions (or $0.57 per dollar invested).
  • There were no other cost-saving measures identified. Some interviewees stated that TGI-4 was a very lean program, and that there were no further cost savings to be made without impacting effectiveness and achievement of results.

5.1.3 What, if anything, could have been done better? Are there any lessons learned?

The Program has applied lessons learned from previous iterations of TGI. TGI-4 is very well-aligned to the role of the federal government as detailed in the IGA. There were certain areas in program design and delivery that were identified as potential areas for improvement. These were: (1) effectiveness of information dissemination activities and/or strategies; (2) better coordination of stakeholder engagement activities and mechanisms; and (3) the G&C process with academic institutions.

  • On the part of stakeholders, the most prevalent comments regarding opportunities for improvement related to the need for better information dissemination and communication. Many stakeholders were unaware of the range of TGI-4 outputs produced to date, or how to access the information. There was a stated desire on the part of some stakeholders to see a more proactive information dissemination strategy.
  • The expanded project planning process during the first year of the Program, along with other external factors, contributed to a delay in program implementation. Some of the delay can be attributed to the lengthy consultative process, and the need, in some cases, to implement consultative mechanisms for stakeholder groups such as the provinces/territories. For example, the TGI-4 Sub-committee of the NGSC was established in March 2011, or almost one year after the announcement of the Program.
  • During the first year of the Program, there were some strains in the relationship between NRCan and academia due to the lapse of the NRCan Terms and Conditions for their G&C program. This caused confusion on the periods to be covered by the grant agreements during the first year of implementation. It was difficult for universities to commit to student research projects without having multi-year funding, and the first TGI-4 grant agreements were only for one year. Almost all academia stakeholders commented on the difficulties that could be posed by delays in transfer of funding.

5.1.4 To what extent have the TGI-4 outputs been produced and at what cost?

The costs of TGI-4 outputs varied widely. There were over 70 distinct research activities being conducted across Canada for a variety of purposes (e.g., mapping, technology development, indicator mineral descriptions, etc.). It was not possible to compare costs across a diverse range of research activities. As examples:

  • The Program has produced approximately 300 outputs at a cost of $21.1M in C-base expenditures (to March 2014). The outputs vary from scientific manuscripts and airborne surveys, to conference abstracts and PowerPoint presentations. There are no cost comparators as each research product is, to some degree, unique. The program does not currently have a systematic means for identifying input costs.
  • The Ni-Cu-PGE-Cr project has produced over 140 outputs, but they vary in their scope, purpose and size and therefore in their cost. As an example:
    • an airborne gravity gradiometer survey at the McFaulds Lake area in Ontario produced nine geophysical maps. The airborne survey was cost-shared with Ontario Geological Survey and cost TGI-4 $1.3M, or $145,000 per output;
    • research on sulphur isotopes produced two papers of between 8-10 pages for $54,000, at a cost of $27,000 per output;
    • research in Turnagain B.C. on Copper and PGE produced one map and one paper of 11 pages at a cost of $105,000, or $52,500 per output.
  • A-base salary costs were not allocated by project as was the case with C-base resources.

5.1.5 What is the difference between planned and actual spending?

There were no major variances in planned and actual expenditure at the program level. Funding (C-base) that was delayed during the initial phase of the Program was reallocated to the last two years of the Program. The Program is expected to be 98% expended by March 2015.

  • The Program managed planned and actual spending well given the delays in program delivery. Funding was re-allocated to the last two years of the Program through the Annual Reference Level Update (ARLU) process. Specifically $2M was re-profiled from 2011-12 to 2013-14 and $1M was re-profiled from 2012-13 to 2014-15. The funding is expected to be almost completely expended by March, 2015.
  • There were some changes in individual project budget allocations during the initial two years of the Program, but the reasons for changes were not always well communicated to project leaders. There is regular financial monitoring of the program, projects, and activities at both the program and project level. These are multi-year projects and allow flexibility to move activities and resources from one fiscal year to another, depending on project requirements and operational realities. This degree of flexibility is to be expected and is a requirement for an iterative project planning approach for multi-year initiatives.
  • Table 2 (Section 1.3) presented a financial summary of C-base budget and expenditure figures to March 31, 2014. Table 6 below provides more detailed expenditure information, by cost centres used by the Program to financially monitor the program activities. In addition to the seven ore body projects and the Methodology project, there were also cost centres established for internal GSC services (e.g., laboratory, cartography and geoanalytical services). Resources were also allocated for Knowledge Management (e.g., information dissemination activities) and TGI-3 legacy activities to finalize products.
Table 6: Actual and Projected Expenditure by Cost Centre
  2010-11 2011-12 2012-13 2013-14 2014-15    
Cost Centre Actual Actual Actual Actual Notional Total %
Coordination 216,705 537,470 285,721 656,215 442,137 2,138,248 9%
TGI-4 Knowledge Management -   24,318 76,701 17,750 -   118,769 0%
TGI-3 Legacy 206,105 13,133 34,557 654 -   254,449 1%
Scientific Laboratory Network 4,895 121,562 272,377 248,813 -   647,647 3%
GSC Digital Cartography -   13,000   15,316 -   28,316 0%
GSC Geoanalytics -   -   -   -   -   -   0%
Methodology -   2,549,023 942,017 1,071,286 647,600 5,209,926 21%
VMS 62,985 118,405 222,228 305,918 152,000 861,536 3%
Ni-Cu-PGE-Cr 1,430,947 416,695 746,154 723,894 200,000 3,517,690 14%
SEDEX 13,774 121,526 358,228 510,832 186,000 1,190,360 5%
Intrusion related 318,293 271,975 508,510 488,224 163,400 1,750,402 7%
Lode Gold 89,732 382,652 1,026,707 1,231,594 569,200 3,299,885 13%
Specialty Minerals 699,420 232,648 95,317 94,461 75,000 1,196,846 5%
Uranium 7,548 100,128 109,892 494,843 240,950 953,361 4%

Sub-total

3,050,404

4,902,535

4,678,409

5,859,800

2,676,287

21,167,435

85%

EBP and Internal Services
           
3,247,131

13%

Total
            
24,414,566

98%

Source: TGI-4 program documentation

5.1.6 Is performance information being collected and used on an ongoing basis?

Performance information was collected and used. Individual projects presented mid-year and end-of-year assessments, outputs were tracked and a special study was contracted to research the attribution of TGI-4 activities to exploration outcomes.

  • There was both program and project performance monitoring. Project progress was reviewed at mid-year and at year-end. During these reviews, project leads presented an assessment of the performance of their project. Standard templates were used and the format aligned to the TGI-4 Logic Model. Basic performance data on outputs were collected and anecdotal information on the uptake of outputs by industry was captured (e.g., comments by industry representatives, press releases). The Program’s performance measurement plan was partially implemented. However, a considerable amount of data collection was scheduled for the final year of the Program.
  • Given the importance of information dissemination to achieving program results, an area identified for improvement was the tracking of basic performance data on information dissemination activities.
  • The Program contracted a study to research the attribution of TGI-4 activities to outcomesFootnote 38. The research builds on previous papers that explored the role of public geoscience and challenges in attributing public geoscience to exploration results. The findings of the study indicated that TGI-4 was likely to achieve its intermediate outcomes.

5.1.7 To what extent can the impacts and effects noted, if any, be attributed to the TGI-4?

In the short-term, TGI-4 has had a direct impact on the supply of trained HQPs that are available for the workforce, and the volume of research that has been conducted on deep mineral exploration. The long-term impacts of TGI-4 are still to be determined.

  • If TGI-4 had not been implemented, the short-term impacts would not have occurred, which would reduce the likelihood of achieving longer-term impacts. In the short-term, there would be fewer HQPs trained to enter the workforce, and less research would have been undertaken. The early success cases of TGI-4 and industry uptake would not have happened.
  • The vast majority of interviewees and survey respondents supported the idea that TGI-4 would have a future impact on deep mineral exploration.

The ultimate measure of exploration success is the cost-effective discovery of an economic mineral deposit. However, this can only be measured ex post facto. Exploration is a staged process, taking place over many years, and different metrics of efficiency and effectiveness may be required at different stages. Moreover, discovery of a given deposit is often the result of exploration by a succession of companies. Although these players may share the same ultimate goal, they likely have different business models and different definitions of success.Footnote 39

6.0 Conclusions

Relevance

  • There is a well-defined need and role for the federal government in the production of public geoscience, and TGI-4 is well aligned to government and NRCan objectives and priorities.
  • NRCan has both the mandate and expertise to deliver programs to address the need for public geoscience. TGI-4’s collaborative approach helps ensure industry needs are being addressed through relevant research that is well coordinated with other initiatives.

Effectiveness

  • The Program has effectively produced a wide range of outputs.
  • The Program's Logic Model was designed to reflect intermediate and ultimate outcomes that go beyond the five-year funding cycle that ends in 2015. There is strong evidence that the Program is making progress in achieving two of its three immediate outcomes within the five-year period.
  • In terms of the achievement of intermediate outcomes, there is evidence of some early uptake by industry of TGI-4 outputs that have influenced exploration activities and strategies.
  • The production and effective dissemination of final synthesis research products will be critical for achieving the expected results of the Program. In order to maximize the impact of these products, information dissemination processes should be strengthened.
  • Given the increasingly prominent role of academia in program delivery, it is important to address outstanding issues regarding consultation and administrative arrangements between NRCan and academic institutions.

Efficiency and Economy

  • The Program is using sound, science-based project management techniques and tools. There is ongoing performance and financial monitoring of the projects and program, but according to the performance measurement plan there are a considerable number of data collection activities to occur in 2014-15.
  • Overall program management costs are reasonable, representing approximately 10% of the C-base program budget. The multi-stakeholder, collaborative approach of TGI-4 produces opportunities for cost-sharing and leveraging of resources. It also presents some opportunities for improvement regarding information dissemination and stakeholder engagement.

7.0 Recommendations

Recommendation 1: Ensure the completion and delivery of high-quality, final products that synthesize the project results.
The production of outputs by the Program has been significant to date, but many of the outputs are preliminary or interim in nature. There are benefits to publicizing information as it becomes available, but the full significance and relevance of the research may only be fully known when all the pieces are assembled in a final synthesis product.

To maximize the results of years of research, it is critical that adequate time and resources be provided to produce and effectively disseminate final products. The challenge is managing the workload of project and activity leads when new programs are starting and requiring significant investment of time on project development, while trying to finalize and disseminate final products. The Program should investigate and assess various options to ensure that project and activity leads have the necessary time and resources to complete final products.

Recommendation 2: Develop an information dissemination and communication strategy and plan that include the collection of performance measurement data.
Information dissemination and knowledge transfer are critical activities for the Program to achieve its expected outcomes yet the program was not tracking performance measure data on these activities. The information has to reach the right people in a format that is accessible and user-friendly. A focused information dissemination and communication strategy and plan should be developed.

Recommendation 3: Develop a clearly articulated stakeholder engagement strategy.
The Program's multi-stakeholder, collaborative approach is paramount for this type of initiative. A scan of similar efforts in other countries and jurisdictions will almost always find the tripartite of government, academia and industry working together. Yet TGI-4 suffered from some delays while appropriate consultative mechanisms were put in place, for example with the provincial and territorial geological surveys (PTGSs). At the same time, industry is a large grouping and dependence on any one forum over another must be carefully assessed. Industry, PTGS and academia are consistent partners of GSC and it is not clear why any new GSC program is required to establish new or special, sub-committees. This takes time during the important start-up phase and can cause delays. It is therefore recommended that a stakeholder engagement strategy be developed to assess the feasibility of establishing semi-permanent, all-purpose stakeholder committees for GSC programs, so as to avoid start-up delays.

Recommendation 4: In any subsequent iteration, develop a Logic Model that is appropriate to the funded period, and continue with special studies to assess long-term outcomes.
The TGI-4 Logic Model identifies outcomes that are only to be achieved after the conclusion of the Program. This acknowledges the longer cycles of research and mineral exploration that are inherent in the sector and this program. It is recommended that in any future TGI iterations, realistic outcomes that can be achieved within the timeframe of the funding cycle be set for immediate and intermediate outcomes. This can present challenges to the GSC given that the strongest argument for supporting public geoscience is that it contributes to discoveries, but this cannot be realistically expected within a five-year funding cycle.

It is also recommended that GSC ensure that the linkages are made to a larger TGI program theory that will enable the single iteration of TGI to provide information on the immediate outcomes of the program, but which would collect cumulative information so that the long-term outcomes can actually be measured. It should continue to commission special studies (e.g., longitudinal studies, contribution analysis, etc.) that can continue to collect evidence and assess the contribution of its public geoscience programs to long-term outcomes, such as improved exploration effectiveness, contribution to discovery of buried ore deposits, or benefits to the economy.

Annex A: TGI-4 Logic Model

Annex A: Logic Model

 
Text version

Annex A: TGI-4 Logic Model

The logic model is organized in a series of activities, outputs, and outcomes that are linked in a chain of results. Completing the Activities and producing the Outputs is expected to result in the achievement of the Immediate Outcomes. Achieving the Immediate Outcomes is expected to result in improved exploration models, the adoption of which will lead to new exploration strategies. The new exploration strategies will lead to an increase in Canada’s metal reserves though discovery of significant buried metal deposits which will ultimately result in high paying jobs in rural communities.

Outcomes Outputs Activities
Long-Term 2016 and Later Intermediate
by April 2016
Immediate
By April 2014
Canada’s metal reserves increase through discovery of significant buried metal deposits High paying jobs in rural communities
  • Knowledge and technology adopted by industry results in new exploration strategies
  • Improved exploration models where important knowledge and information gaps have been addressed.
  • Network of Canadian research centers focused on relevant exploration research problems
  • Improved understanding of distal indicators to ore environments.
  • New methods and technologies allow better detection of buried ore deposits are disseminated to industry
  • Compilations of existing data for target districts
  • Analytical protocols to characterize mineral systems
  • Descriptions of enhanced understanding of mineral systems
  • Geochemical and mineralogical indicators targeting ore environments
  • Data dissemination though workshops, seminars, presentations and publication
  • Technology suitable for implementation by industry
  • HQP with suitable experience for industry
  • Analysis  and prioritization of  research to improve industry standard exploration models
  • Compilation and analysis of existing data from target areas
  • 3D physical and chemical modeling of deposit  environments
  • Development of robust indicators for ore forming environments
 

Annex B: Evaluation Matrix