Audit of NRCan’s Publishing Activities (AU1706)
Audit and Evaluation Branch
Natural Resources Canada
Presented to the Departmental Audit Committee (DAC)
December 14, 2017
Table of Contents
Executive summary
The Government of Canada (GoC) uses publications to communicate policies, programs, scientific information, services, and initiatives to the Canadian public and industry stakeholders. The GoC’s Procedures for Publishing defines a publication as “an information product with a long shelf-life produced by or on behalf of the GoC in any medium or format.” Natural Resources Canada’s (NRCan) publishing activities are governed by a number of government-wide and departmental policies that call for publications to contain timely, accurate and accessible information in a variety of traditional and digital formats and to be produced in an efficient and cost-effective manner. Open Government initiatives and Open Science commitments further require that departments maximize the release of information and data of business value in order to support transparency, accountability, and citizen engagement.
As part of Canada’s Open Government commitments, federal science-based departments and agencies will be required to make the data supporting the observations and conclusions of their science and technology (S&T) publications publicly available. Open Government and Open Science initiatives are challenging federal science based departments and agencies to rethink the way they disseminate their S&T publications and communicate their science to a broader audience. Therefore, Open Government and Open Science efforts to increase the availability of federally funded science and technology research and to engage the public in that research is an important consideration for the Department in its approach to S&T publishing.
As a science based department, NRCan invests significant resources in its S&T activities. In 2016-17, NRCan spent $595 million on S&T and well over half of its workforce was engaged in S&T activities. Many of these activities involve research projects that lead to results or observations that need to be communicated in order to help advance knowledge in specific areas, address departmental priorities, support regulations, and explore economic opportunities. In many instances, these results are communicated to informed audiences through S&T publications, defined as “any material or content conveying S&T information (beyond raw data), prepared for publication, regardless of publishing medium, and produced for an informed and/or specialised audience (i.e. not broad public dissemination)."Footnote 1 These could be books, peer-reviewed journal articles, maps, and technical reports. The Lands and Minerals Sector and the Canadian Forest Service produce the vast majority of NRCan’s S&T publications. In 2016-17, the Department published approximately 1200 S&T articles and hundreds of other research-related documents.
NRCan also produces a variety of non-S&T publications. Most of these are managed by the Communications and Portfolio Sector, and are governed by NRCan’s Publishing and Exhibits Policy. This policy defines publications as “an entity on or in which information (intellectual content) is written, recorded, stored or reproduced; is capable of being read or otherwise perceived; and issued to the public with or without charge or to an internal audience.Footnote 2Many of these publications are science related, but they are targeted at a broader audience and the language used is less technical than that of scientific publications. Non-S&T publications include items targeted at external audiences such as publications posted on the department’s website, pamphlets, fact sheets, corporate videos and promotional items used at exhibits and conferences. NRCan produced approximately 200 of these types of publications in both fiscal year 2015-16 and 2016-17.
The objective of the audit was to assess the effectiveness of governance and management processes in support of NRCan’s publishing activities.
Strengths
The Department has developed mechanisms to help align its S&T and non-S&T publishing activities with the Department’s mandate and its program activity architecture. It has also developed tools to facilitate the broad dissemination of S&T publications produced by its two largest S&T Sectors so that federally funded science carried out by these Sectors is widely available.
Areas for improvement
Opportunities exist to strengthen the governance of NRCan’s S&T and non-S&T publishing. In particular, efforts to revise the current S&T Publications Policy should focus on clarifying the responsibilities and accountabilities of the Office of the Chief Scientist. There are also opportunities to strengthen and harmonize some of the Department’s S&T publishing tools so that all sectors are able to disseminate their S&T publications effectively. NRCan has not periodically revisited its approach to disseminating S&T publications and its mechanisms for assessing whether publishing activities are meeting the needs of intended audiences. Efforts in these areas have been limited to sector-specific initiatives and a recently completed pilot project on research intelligence tools.
Internal audit conclusion and opinion
NRCan has adequate governance processes for its S&T and non S&T publication, but they could be further strengthened by clarifying roles and responsibilities for S&T publications. The development and implementation of more uniform guidance and publication tools would also help the Department better manage the planning, monitoring and reporting of its publications and contribute to assessing whether its publications are meeting the needs of intended audiences.
In my opinion, the complex governance and decentralized nature in place for the management of publishing activities will require the involvement of all sectors in implementing the recommendations contained in this report in order to respond to the evolving nature of publishing in the context of Open Government and Open Science.
Statement of conformance
In my professional judgement as Chief Audit and Evaluation Executive, the audit conforms to the Institute of Internal Auditors' International Standards for the Professional Practice of Internal Auditing and the Government of Canada’s Policy on Internal Audit, as supported by the results of the Quality Assurance and Improvement Program.
Christian Asselin, CPA, CA, CMA, CFE
Chief Audit and Evaluation Executive
December 14, 2017
Acknowledgments
The audit team would like to thank those individuals who contributed to this project and particularly employees who provided insights and comments as part of this audit.
Introduction
The Government of Canada (GoC) uses publications to communicate policies, programs, scientific information, services, and initiatives to Canadians and industry stakeholders. The GoC’s Procedures for Publishing defines a publication as “an information product with a long shelf-life produced by or on behalf of the GoC in any medium or format.” Publishing activities have evolved in the last few years to reflect the changing communication preferences of Canadians, the advancement of digital communication technologies, and the GoC’s shift to more environmentally friendly operations.
Natural Resources Canada’s (NRCan) publishing activities are governed by a number of government and departmental policies. In sum, these policies require departments to publish timely, accurate and accessible information in a variety of traditional and digital formats and to do so in an efficient and cost-effective manner. The Government of Canada’s Third Biennial Plan to the Open Government Partnership requires that departments maximize the release of information and data of business value in order to support transparency, accountability, and citizen engagement.
As part of Canada’s Open Government commitments, federal science-based departments and agencies will be required to make the data supporting the observations and conclusions of their S&T publications publicly available. Open Government and Open Science initiatives are challenging federal science based departments and agencies to rethink the way they disseminate their S&T publications and communicate their science to a broader audience. Therefore, Open Government and Open Science efforts to increase the availability of federally funded science and technology research and to engage the public in that research is an important consideration for the Department in its approach to S&T publishing.
As a science based department, NRCan invests significant resources in its science and technology (S&T) activities. In 2016-17, the department spent $595 million on its S&T activities and well over half of its workforce was in engaged in S&T activities. Many of these activities involve research projects that lead to results or observations that need to be communicated in order to help advance knowledge in specific areas, address departmental priorities, support regulations, and explore economic opportunities. In many instances, these results are shared through S&T publications, defined as “any material or content conveying S&T information (beyond raw data), prepared for publication, regardless of publishing medium, and produced for an informed and/or specialised audience (i.e. not broad public dissemination)."Footnote 3These could be books, peer-reviewed journal articles, maps, and technical reports. The Canadian Forest Service and the Lands and Minerals Sector produce the vast majority of these publications for informed audiences. In 2016-17, the Department published approximately 1,200 S&T articles and hundreds of other research-related documents. All of these publications are required to adhere to NRCan’s Science and Technology (S&T) Publications Policy which is overseen by the Office of the Chief Scientist (OCS).
NRCan also produces a variety of non-S&T publications. Most of these are managed by the Communications and Portfolio Sector, and are governed by NRCan’s Publishing and Exhibits Policy. This policy defines publications as “an entity on or in which information (intellectual content) is written, recorded, stored or reproduced; is capable of being read or otherwise perceived; and issued to the public with or without charge or to an internal audience.”Footnote 4 These publications are targeted at a broader audience and include such things as non-technical publications posted on the Department’s website, pamphlets, fact sheets, corporate videos and promotional items used at exhibits and conferences. A specific example is the Department’s annual State of the Forests Report. NRCan produced approximately 200 of these publications in both fiscal year 2015-16 and 2016-17.
Audit purposes and objectives
The objective of this audit was to assess the effectiveness of governance and management processes in support of NRCan’s publishing activities.
Specifically, the audit assessed whether:
- NRCan had established adequate processes to enable effective and efficient publishing activities that align with the Department’s mandate and strategic priorities and complied with applicable policies; and
- NRCan effectively managed its publishing activities by ensuring that its publications met the needs of its intended audiences.
Audit considerations
A risk-based approach was used in establishing the objectives, scope, and approach for this audit engagement. A summary of the key inherent risks that could impact the effective management of publishing activities include:
- NRCan’s scientific publishing activities may not align with the Department’s mandate or contribute to advancing its strategic priorities;
- Department-wide governance and management processes for scientific publishing may not be effective, leading to inefficiencies (e.g. duplication of efforts, lack of coordination), and non-compliance with GoC and departmental policies;
- NRCan’s publishing activities may not effectively meet the needs of their intended audiences; and
- NRCan’s publications may not be effectively disseminated within the Department (e.g. effective policy-science integration) and to stakeholders (industry, academia etc.).
Scope
The scope of the audit included a review of the Department’s publishing activities from April 2015 to March 2017, and focused on both scientific and non-scientific publications subject to NRCan’s Publishing and Exhibits Policy and its Science and Technology Publications Policy.
The audit did not examine other communication products such as news releases, media advisories, advertising, or web content that requires real-time maintenance, as these are not considered publications as per GoC and NRCan policies.
The scope of the audit did not include the management of intellectual property because an audit of this area was completed in March 2017. Although this audit identified specific challenges regarding the management of publications in the context of Open Government, it did not examine the extent to which the Department is meeting Open Government requirements. An NRCan internal audit of Open Government is planned for 2019-20.
Approach and methodology
The approach and methodology followed the Institute of Internal Auditors' International Standards for the Professional Practice of Internal Auditing and the Government of Canada’s Policy on Internal Audit. These standards require that the audit be planned and performed in such a way as to obtain reasonable assurance that audit objectives are achieved. The audit included tests considered necessary to provide such assurance. Internal auditors performed the audit with independence and objectivity as defined by the International Standards for the Professional Practice of Internal Auditing.
The audit approach included the following key tasks:
- Interviews with key personnel; and
- Review of key documents, business processes, and relevant background information.
The audit also consulted with four federal science-based departments and agencies to obtain an understanding of how they govern and manage their scientific publishing activities.
The conduct phase of this audit was substantially completed by June 2017.
Criteria
The audit criteria were developed primarily from the key controls set out in the TBS’ Core Management Controls, the Management Accountability Framework, and relevant associated policies, procedures, and directives. Other sources included the Government of Canada’s Policy on Communications and Federal Identity and Procedures for Publishing, as well as the NRCan’s Science and Technology Publications and Publishing and Exhibits policies. The criteria guided the fieldwork and formed the basis for the overall audit conclusion. Please refer to Appendix A for the detailed audit criteria.
Findings and recommendations
Managing Publishing Activities
Summary Finding
The audit found that NRCan had some processes in place to help ensure that its publishing activities were aligned with its mandate and priorities. Opportunities were identified to improve the effectiveness and efficiency of its S&T publishing activities by strengthening its governance of the publishing process and by developing and implementing, more uniform guidance and publication tools. Improvements in these areas would support NRCan’s compliance with policy requirements and increase the accessibility of its S&T publications. Opportunities were also identified to improve the planning of non-S&T publications by taking steps to develop more realistic publishing plans that result in fewer cancellations.
Supporting Observations
The audit expected NRCan to have governance structures that adequately supported the management of its publishing activities and to have established the related planning, monitoring, and reporting processes to ensure that publications are aligned with the Department’s mandate and priorities. The audit also expected the Department to have provided sufficient guidance and tools to support the effective and efficient management of publishing activities so that these activities would comply with Government of Canada and departmental publishing policies.
Governance
The Government of Canada’s Policy on Communications and Federal Identity and the related Procedures for Publishing provide direction to federal departments and agencies on the management of their communication and publishing activities. Natural Resources Canada’s Publishing and Exhibits Policy was issued in accordance with these policies and it governs the Department’s publishing activities. The Department’s policy’s objective is to ensure that the publishing of items such as promotional materials, corporate videos and publications intended for general audiences is well coordinated, effectively managed and responsive to the information needs of the public. The Communications and Portfolio Sector (CPS) coordinates the non-S&T publications process for the Department in collaboration with the other Sectors.
NRCan’s Science and Technology (S&T) Publications Policy governs the Department’s S&T publishing activities and it is embedded as an appendix in the broader Publishing and Exhibits Policy. The policy’s objective is to facilitate access to NRCan’s S&T publications, increase their impact and ensure that the Department is able to respond to Open Government requirements. The Office of the Chief Scientist is responsible of overseeing the policy, but each Sector manages its own S&T publishing processes.
In accordance with the aforementioned government policies, NRCan’s Publishing and Exhibits Policy identifies the Director General of the Engagement and Digital Communications Branch in the CPS as the Head of Communications. As per the Procedures for Publishing, the Head of Communications is responsible for monitoring the Department’s compliance with the publishing procedures and for reporting on compliance when requested by the Treasury Board. The audit found that NRCan’s Publishing and Exhibits Policy excludes S&T publications from its scope and assigns responsibility for ensuring that these publications comply with the Procedures for Publishing to NRCan’s Sectors and regions. Nonetheless, the Head of Communications remains accountable to Treasury Board for ensuring that all of the Department’s publications comply with the Procedures for Publishing. The audit found, however, that there are no monitoring or reporting mechanisms for the Head of Communications to obtain this assurance. The CPS noted that it does not have the capacity to oversee the compliance of S&T publications and that it does not liaise with the Office of the Chief Scientist (OCS) or the S&T Sectors in this regard.
The audit reviewed NRCan’s Science and Technology (S&T) Publications Policy and found that it does not establish clear roles and responsibilities for the governance of S&T publications. The Office of the Chief Scientist (OCS) is responsible for overseeing the policy, but the policy is not clear on what this entails. The policy, for example, does not authorize the OCS to monitor or gather periodic reports from the science sectors on adherence to the policy or to periodically monitor the S&T publishing practices of other science based departments and agencies in order to identify good management practices that could be adopted by NRCan. The audit also found that the policy states that sector ADMs are accountable for their sector’s adherence to the policy, but does not refer to any related guidance that specifies how they are to do this. Further, S&T publications must respect the Government of Canada’s Policy on Communications and Federal Identity as well as the supporting Procedures for Publishing, but the department’s S&T Publications Policy does not refer to this policy or set of procedures.
The audit also found that there are no requirements or systems in place to ensure that the NRCan Library is regularly updated on the Department’s publishing activities. This may impact the Library’s ability to effectively serve as the Department’s Open Government Champion and to consolidate a definitive collection of the Department’s non-S&T and S&T publications that is readily available to all NRCan employees. The audit noted that one science based department requires that an electronic copy of all S&T publications be submitted to its library so that library staff can help facilitate their dissemination.
In November 2015, the Department’s Director General Science and Technology Committee (DGSTC) approved revisions to the policy goal of the S&T Publications Policy to reflect open government requirements. In March 2016, NRCan’s S&T Publications Policy Working Group was launched to explore further revisions to the Department’s S&T Publications Policy and to facilitate the development of department-wide publishing solutions. At the time of the audit, the working group had made limited progress in revising the policy and in advancing the development of new department-wide S&T publishing tools. The audit further found that the working group had no terms of reference, no stated deliverables, no formal reporting mechanisms to senior management and no documented timelines to complete its work. In acknowledgement of these shortcomings, the DGSTC endorsed a proposal in June 2017 to reorganize the working group as part of a broader restructuring of NRCan’s governance of Open Science. The proposed governance framework includes a formal mechanism for a new ad hoc S&T publications task team to report to senior management and it calls for the team to be provided with a formal mandate, defined deliverables, and timelines for its work.
Managing Non-S&T Publications
The CPS is responsible for producing an annual departmental publishing plan for non-S&T publications. At the beginning of each fiscal year, the CPS requests sector publication coordinators to submit publication proposals. The CPS assesses these proposals to determine, among other things, whether they are consistent with GoC and NRCan priorities. Following its review, the CPS develops a publication planning dashboard with cost estimates for each sector’s senior management approval. The sectors are responsible for notifying the CPS when they want to initiate production of a planned publication and they are responsible for providing the CPS with the publication’s content. The CPS is also responsible for supporting sectors in the production of publications by providing, for example, editorial and graphic design services.
The audit found that in both 2015-16 and 2016-17, sectors cancelled approximately 30% of planned publishing products. In 2016-2017, CFS and LMS (ESS) cancelled more publications than they produced. Sector publication coordinators indicated that the high cancellation rate might be due to the development of sector publication plans that are more of a “wish list” than a realistic projection of the publications that the sector could produce within the year. The audit team shared this observation with management and CPS noted that it is working to reduce the cancellation rate by working with Sectors to improve on-time starts for projects and by better educating the Sectors about the time needed to complete a publication.
The audit found that the CPS tracks the timeliness of its publishing activities by comparing estimated task days versus the actual task days required to complete a publication. CPS’ tracking data indicated that in 2016-17 it took, on average, three times as many task days to complete a publication than expected. CPS told auditors that there can be a variety of reasons for this, including delayed approvals from sector management, the late submission of publication text, and ongoing exchanges between the sectors and the CPS over editorial changes. Due to the limitations of the publication tracking system, however, CPS is not able to systematically analyse its production process to determine whether there are recurring delays at particular points in the process and to identify specific actions that could be taken to reduce these delays.
Managing S&T Publications
Workflow tools
The monitoring of S&T publications as they evolve from publication proposals to conference presentations, internal reports or externally published peer reviewed articles is carried out within the sectors and through the use of Sector-specific workflow systems. These various systems are designed to help ensure that publications are of high quality, obtain the necessary approvals, and comply with policy requirements. Some of these systems are fully automated whereas others are a mix of electronic and paper-based processes. The audit found that all sectors have established publication approval processes within their workflows that help to ensure that S&T publications align with the Department’s mandate and program activity architecture.
There are benefits to a single department-wide workflow for S&T publications and NRCan has tried to develop one in the past. In 2014, the Director General Science and Technology Committee endorsed GCDOCs as the Department’s single publications workflow and as means of ensuring that S&T publications were compliant with NRCan’s S&T Publications Policy. Sectors were directed to use this workflow to manage the production of their S&T publications. Despite this direction, the Department’s two largest publishing sectors were hesitant to do so because they found the system unresponsive to their publishing needs. Sectors that adopted GCDOCS struggled with the system and authors found it not to be user-friendly. To deal with these challenges, the audit found that sectors developed workarounds and supplementary tracking systems. One sector in particular, makes use of GCDOCS and two other systems to manage its publication tracking. Another sector noted that some of its authors opted not to use GCDOCS at all. “Work-arounds” and the avoidance of workflow tracking systems increases the risk that publications will be non-compliant with publishing policies and they may add additional complexity and cost.
Interviewees noted that the development of a single workflow system for all sectors is a challenging task, but that there is a need and support for such a solution. Sectors using GCDOCS told the audit team that they need a less cumbersome system that helps make their publications more readily discoverable and accessible by NRCan employees and the public. The audit found that the largest publishing sector has long faced challenges in maintaining its increasingly unstable automated workflow and that concerns are growing about system failure and the loss of information. The Sector would like to see the development of a department-wide publication tool accelerated so that it can avoid adopting another sector specific solution. NRCan’s 2017-2020 Information Technology Plan includes a planned expenditure of $25,000 to identify a potential department-wide publication solution, but is uncertain whether this initiative will be completed in time to meet the needs of the Sector.
Publication review
In order to maintain the quality of NRCan’s S&T publications, the S&T Publications Policy requires that all S&T publications undergo peer-review. For internally published publications, this requires that the material to be published be reviewed by NRCan employees who are knowledgeable in the area, but who are not the authors of the publication. In the case of publications that are published externally in academic journals, the Department relies on the peer review processes of the respective journal and only requires that a publication be reviewed within the Department for non-science content.
The audit found that each sector’s publication workflow included a peer review or internal review requirement for internal publications, but that the nature of this review and the extent to which it was documented varied across and within sectors. For example, in one Sector there are no formal internal review tools; whereas another sector has developed guidelines for its internal reviewers and a review checklist to increase the rigour of the review process. Some interviewees expressed an interest in being provided with high-level guidance from the OCS in the form of “peer review principles” which could help clarify the policy’s expectations for various types of publications and help foster a more consistent approach within the Department.
“Gold publishing” payments
In 2016-17, approximately 500 NRCan S&T publications were published in external peer-reviewed journals. Externally published research is often subject to an “embargo” period of usually one year, but some journals have embargo periods that can be two years or longer. During this period, the external publisher or journal controls access to these publications and those wishing to access them must either have a paid subscription to the journal or purchase access to the individual article. Some external publishers charge “gold publishing” fees, which are paid to forego an embargo period and to make a publication immediately available to the public free-of-charge. These fees can range from $500 to $5000. The payment of ‘gold’ publishing fees is one approach the Department can use to ensure that its S&T publications are open access at the earliest possible opportunity. There are, however, risks regarding the payment of “gold publishing” fees that concern the Department. One of these risks relates to the reputation of some publishers who charge these fees and conduct inadequate peer-review. Another concern is that “gold publishing” fees reduce funding available for other projects and may not represent good value for taxpayers.
The Department has produced guidance that discourages the payment of gold publishing fees to obtain “open access”, but leaves it to the discretion of Sector management to authorize these payments on a case-by-case basis. One sector assessed its use of open access publishing between 2013 and 2015 and found that approximately 12% of its external publications were originally published as open access. The assessment did not indicate whether any gold publishing fees were paid. Despite the concerns identified by the Department, the audit found that the current S&T Publications Policy does not address the issue of paying gold publishing fees and there is no ongoing department-wide or sector-wide monitoring of these payments to analyse trends or to justify their value.
Reporting on publishing activities
As previously noted, the production of publications is managed by a several publication workflow systems and processes vary across various sectors, branches, regions and labs. These factors pose challenges for the Department in complying with Treasury Board requirements pertaining to the periodic reporting of its S&T and non-S&T publishing activities. As required by the GoC’s Procedures for Publishing, the audit found that CPS provides an index of non-S&T publications that it has produced to Public Services and Procurement Canada (PSPC) and Library and Archives Canada (LAC). Some of the S&T Sectors also provide an index of their publications to these organizations. However, not all S&T sectors report this information and the indices that are provided are not coordinated or verified to be an accurate and comprehensive listing of the Department’s publishing activity. A coordinated department-wide approach to tracking publications on an ongoing basis and for ensuring that a departmental index is provided to PSPC and LAC may help address these challenges.
The audit found that the OCS has taken some steps to compile a report of forthcoming S&T publications identified by the Sectors as being of interest to senior management. This report is used by the Chief Scientist to brief the Deputy Minister and the Executive Committee. In some cases, it has also been used to identify a publication that might benefit from a broader communication strategy that may help reinforce its findings and further its dissemination. Although the production of this report is a positive step, it does not provide a comprehensive real-time picture of NRCan’s S&T publishing activities.
Risk and impact
The absence of clear roles, responsibilities, and accountabilities for the governance of the Department’s publishing activities increases the risk that publications will not comply with applicable policies and that the Department may lack the tools needed to strengthen the monitoring and reporting of its publishing activities.
Recommendations
- It is recommended that the Assistant Deputy Minister (ADM), Communications and Portfolio Sector (CPS) and the Chief Scientist revisit the Publishing and Exhibits Policy to clarify how the Head of Communications can obtain the information needed to fully account to the Treasury Board for the Department’s science and technology (S&T) publishing activities.
- It is recommended that the ADM, Communications and Portfolio Sector and the Chief Scientist clarify the roles and responsibilities of all parties involved in publishing activities.
Other findings from this section pertaining to the management of publishing activities are addressed in recommendation 4 on page 13.
Management response and action plan
Management agrees. In response to recommendation 1:
NRCan’s Open Science Committee is creating a task team to review the NRCan Publishing and Exhibits Policy and the NRCan S&T publishing Policy (see recommendation 4). This review will be completed by June 2018 and will include a detailed review of roles and responsibilities of all parties involved in publishing activities.
A member of CPS will be part of this task team to ensure that it clarifies how the ADM of CPS will be provided with all the required information to report to TBS.
Position responsible: ADM, Communications and Portfolio Sector and Chief Scientist
Timing: June 2018
Management agrees. In response to recommendation 2:
Once the revisions to the NRCan Publishing and Exhibits Policy, and the S&T Publication Policy are finalized, the Chief Scientist and the ADM of CPS will meet to review the updated policy and validate that all the proper mechanisms, roles and responsibilities are in place to ensure that NRCan can fully account for its publishing activities.
Position responsible: ADM, Communications and Portfolio Sector and Chief Scientist
Timing: September 2018
Meeting Stakeholder Needs
Summary Finding
The audit found that NRCan has systems in place to disseminate most of its S&T publications. The Department’s smaller publishing Sectors, however, face some challenges making their publications readily discoverable and accessible by NRCan employees, stakeholders and the public. The audit also found that the NRCan has not revisited its approach to S&T publishing in the context of Open Government and that mechanisms for assessing whether publishing activities are meeting the needs of intended audiences are not fully developed. To date efforts in this area have been limited to sector-specific initiatives and to a small pilot project that has recently been completed.
Supporting Observations
The audit expected NRCan to have processes in place to effectively disseminate its publications within the Department and to stakeholders. The audit also expected NRCan to periodically revisit its approach to disseminating and promoting its publications in order to satisfy the expectations of an evolving publishing environment, and to establish measures to assess the extent to which its publishing activities are meeting the needs of its intended audiences.
Dissemination tools
The effective dissemination of S&T publications is critical to ensuring that federally funded research is discoverable by, and accessible to, NRCan employees, external stakeholders and members of the public. The Department’s current S&T Publications Policy is focused on making S&T publications readily available to its employees and senior management in order to support the integration of policy and science and to advance evidence based decision-making.
In May 2016, NRCan adopted the Federal Science Library (FSL) as its new single online point of access for NRCan publications. The FSL is an online portal that provides an index of print collections and repositories of seven science-based departments and agencies. It is not a repository for publications but it is an important tool that can help the public discover the Department’s publications. The NRCan Library, within Corporate Management and Services Sector has been working with sectors to link existing Sector publications databases with the FSL. The audit found that efforts to do this for the Department’s smaller publishing Sectors were hindered by their publication workflows and their lack of a publications database. As a result, publications from these sectors may not be well represented or easily discoverable in the FSL. The audit also found that there are no mechanisms to ensure that updated information pertaining to these sectors’ publications is transmitted to the FSL on an ongoing basis. Although these sectors do not generate as many S&T publications as larger publishing Sectors, it is important that their publications be discoverable by stakeholders and the public. The Department’s largest publishing Sector has established a direct link between its database and the FSL to ensure that updated information on its publications is provided on an ongoing basis. The Department’s other high volume publisher has explored the development of such a link, but continues to use a manual process and updates the FSL monthly.
As noted, the FSL is not a repository for NRCan publications and anyone trying to access the Department’s publications through the FSL will be redirected to websites of external publishers or journals, and the publication databases of the Department’s two largest S&T publishers. Unlike several other science based departments and agencies, NRCan does not have a single department-wide repository to house all of its publications and this makes it difficult to ensure external and internal access to these publications. For example, without a repository the Department does not have a means to store and share “embargoed” publications so that NRCan employees can easily access them without payment. Additionally, Sectors without publication databases face significant challenges in making their S&T publications widely accessible within the Department and to the public.
Disseminating publications in an evolving environment
Open Government and Open Science initiatives are challenging federal science based departments and agencies to rethink the way they disseminate their S&T publications and communicate their science to a broader audience. As previously noted, one of the objectives of these initiatives is to increase the public availability of federally funded science and technology research.
In order to promote effective communication and to share new knowledge of S&T results with NRCan employees and others who are not subject matter experts, the Department’s S&T Publications Policy requires that plain language summaries and policy relevance statements be prepared for all S&T publications. The audit found that almost all sector workflows include this requirement, but that no sector periodically assesses the quality of its plain language summaries. The audit also found that there has been no department-wide assessment of the extent to which plain language summaries and policy relevance statements are used by policy analysts or other NRCan staff to strengthen the integration of science and policymaking. Some interviewees noted that the summaries are too technical and that there are concerns about the extent to which they are used. In the smaller publishing sectors, the audit found that plain language summaries and policy relevance statements are not searchable or easily accessible by those outside of the Sector. The audit noted that one science based department has incorporated the communication and dissemination of plain language summaries to senior management into its policy and procedures for scientific publishing. This includes identifying internal clients (other sectors, branches, or regions responsible for the potentially impacted programs, projects, Acts, initiatives) and ensuring that those clients and the Communications Branch receive plain language summaries.
The audit found that the extent to which NRCan Sectors comply with the requirement to produce plain language summaries and policy relevance statements varies. Some Sectors consistently produce these summaries and statements, but others have not followed this practice. This may be attributable to uncertainty as to whether plain language summaries and policy relevance assessments are required for all S&T publications. For example, publishing guidelines in one sector note that plain language summaries and an assessment of policy relevance are only required for publications having “contentious issues.” In another Sector, a policy relevance assessment is not a mandatory step in the publication workflow.
As part of Canada’s Open Government commitments, federal science-based departments and agencies will be required to make the data supporting the observations and conclusions of their S&T publications publicly available. This is being done to encourage the reuse of scientific data. A key enabler for this initiative is the use of digital object identifiers (DOI), a unique alpha-numeric code which links publications to related datasets. One of NRCan’s largest publishing sectors has assigned DOIs to its publications for several years, but other Sectors have yet to do this. The OCS and the Chief Information Officer and Security Branch (CIOSB) are leading efforts to finalize a corporate agreement with a service provider to assign DOIs to all of NRCan’s S&T publications and supporting datasets. Once finalized, the Department will need to ensure that guidance on implementing DOIs is provided to all Sectors and that Sector publication workflows are updated to reflect this new publishing requirement.
Revisiting approaches to publishing
Open Government and Open Science efforts to increase the availability of federally funded science and technology research and to engage the public in that research is an important consideration for the Department in its approach to S&T publishing. In 2016-17, approximately 50% of NRCan’s publications were published in external peer reviewed journals and 50% were published internally using a variety of publication types.
Each type of S&T publication has its advantages and disadvantages. External publications – particularly those appearing in high impact journals – may reach a larger, more informed audience and can add credibility to the Department’s research and enhance the reputation of its scientists. Publishing externally, however, often takes much longer (as long as 24 months) than some internal publishing options and external publications may only be available at a cost for the first year (or longer), unless arrangements or payments have been made for immediate open access. Some internal publications are similar to externally peer-reviewed publications, but others are less formal, subject to less review and designed to expedite the publication of research information. All internal publications are also immediately “open access” at no cost to the public.
One of the Department’s larger publishing sectors has recently reviewed the types of S&T publications it has produced over the past several years and noted a reduction in the number of external peer reviewed publications and formal internal publications it produced. The review noted that this reduction corresponded with an increase in a less formal publication series that expedites the release of research information. According to the sector, the need to disseminate research results more quickly to programs and stakeholders may be driving this change. These less formal internal publications are not reviewed to the same extent as external peer-reviewed publications and they may not be appropriate for communicating all of the Department’s S&T research information. Nonetheless, the sector’s analysis indicated that this less formal and timely internal publication series is among its most frequently downloaded publications.
A department-wide review of S&T publishing may help identify publishing trends and ensure that the Department’s approach to publishing considers the benefits associated with the various types of publications (external and internal) and achieves an optimal mix of publication types that meets the needs of its intended audiences. Additionally, a standardized rigorous review process for internal publications can help ensure the quality of internally published science and research.
Reports from cost-recovered research
The audit noted that some Sectors undertake cost-recovery work and produce reports related to this work for industry and other clients. There is some concern within the Sectors as to whether these documents are being consistently managed using publication workflow systems and stored in manner that facilitates their accessibility. Although these reports are considered business protected documents and are currently exempt from Open Government requirements, they can be publicly disseminated after a three to five year period if their content is deemed valuable for general interest and the client consents. In the spirit of Open Government it would be appropriate for the Department to seek this consent at the appropriate time and to disseminate these reports if this consent is obtained. However, there is currently no process to track the embargo period, and to ensure that clients are approached about making these publications publicly available at the end of that period.
Meeting the needs of intended audiences
Considering the Department’s significant S&T expenditures and the volume of its publishing activities, it is important that NRCan have mechanisms in place to determine whether its S&T and non-S&T publications are relevant, impactful and meeting the needs of intended audiences.
The audit found that some steps have been taken to assess whether the Department’s publications are meeting the needs of intended audiences. For non-S&T publications, the Department has developed tools to analyse the use of publications that are in HTML format and to track downloads of documents in PDF format that are posted on the NRCan website. The audit found, however, that these tools are not being used. Some S&T Sectors review the relevance of their S&T publications through regular meetings with stakeholders and through research planning activities that incorporate stakeholder feedback. Surveys have also been used for this purpose. The Department also engaged a third party to conduct an assessment of NRCan’s output of external peer reviewed publications between 2006 and 2014. This assessment focused on, among other things, assessing the extent to which these publications were contributing to specific fields of research as evidenced by citations in other peer-reviewed journals.
Although this initiative is noteworthy, it was not intended to examine the use and impact of the Department’s S&T publications and non-S&T publications (external and internal) on other audiences such as industry, policy-makers and the public.
The audit noted that the Department is currently conducting a pilot project of bibliometric tools. These tools make use of digital object identifiers and are capable of tracking in real-time the reach and impact of the Department’s publications, such as the extent to which the Department’s publications are being referenced in policy documents, the news media, in social media and by academics. Non S&T publications, such as infographics or videos, can also be assigned digital object identifiers and tracked to assess the reach an impact of these publications and whether they are meeting client and stakeholder needs. This information could be further analyzed by the Department to identify subject areas, research scientists and the types of publications that receive the most attention. Such an analysis may help the Department make better informed decisions about, for example, the types of publications it should produce and whether there are significant benefits associated with certain approaches to publishing, such as incurring additional costs for “open access.”
Risk and impact
Each year NRCan produces approximately 1200 S&T publications that communicate the results and observations from the Department’s significant S&T investments. If these publications are poorly disseminated and their impact not measured, the Department’s ability to demonstrate a return on its investment of public funds may be compromised.
Recommendations
- It is recommended that the Assistant Deputy Minister (ADM), Communications and Portfolio Sector (CPS) and the Chief Scientist, in collaboration with the Chief Information Officer and Security Branch (CIOSB), work with the Sectors to develop and implement a department-wide approach to disseminating publications that facilitates the discoverability and accessibility of publications produced by all Sectors.
- It is recommended that the Chief Scientist work with the science and technology (S&T) Sectors to undertake a review of NRCan’s approach to S&T publishing, including updating the S&T Publications Policy. This review should include:
- defining the purpose and objectives of the Department’s S&T publishing activities;
- improving the processes for the management of S&T publishing activities (e.g. approval processes, peer review, the use of plain language summaries); and
- addressing the different ways the Department communicates and disseminates science, and assesses whether the Department’s S&T publications are meeting the needs of intended audiences.
Management response and action plan
Management agrees. In response to recommendation 3:
The Chief Scientist and the CIO, with the support of NRCan Open Science Committee, will work on defining the business requirements to support the dissemination of NRCan’s publications (March 2018) and pending funding, implement of an enterprise solution (March 2019).
Position responsible: Chief Scientist and CIO
Timing: Definition of business requirements – March 2018
Implementation of enterprise solution – March 2019 (pending funding)
Management agrees. In response to recommendation 4:
The Chief Scientist, with the support of NRCan Open Science Committee, will review NRCan’s approach to S&T publishing (including the S&T Publication Policy), notably in light of the Open Government/Open Science initiative, the Policy on Scientific Integrity, the ‘Open by Default’ pilot and NRCan’s IM policy. This review will include the three points identified in recommendation 4 (June 2018).
Following the endorsement of the revised Policy, Chief Scientist will develop a change management plan to ensure people involved are aware of the changes and of their responsibilities (March 2019).
Position responsible: Chief Scientist
Timing: March 2019
Appendix A – Audit criteria
The criteria were developed from the key controls set out in the Treasury Board of Canada’s Core Management Controls and relevant associated policies, procedures, and directives. The criteria guided the fieldwork and formed the basis for the overall audit conclusion.
The objective of this audit was to assess the effectiveness of governance and management processes in support of NRCan’s publishing activities.
The following audit criteria were used to conduct the audit:
| Audit Sub-Objectives | Audit Criteria |
|---|---|
| Audit Sub-Objective 1:
To determine whether NRCan has established adequate processes to enable effective and efficient publishing activities that align with the Department’s mandate and strategic priorities and comply with applicable policies. |
1.1 It is expected that NRCan’s governance structure adequately supports the management of its publishing activities. |
| 1.2 It is expected that NRCan has established planning, monitoring, and reporting processes to ensure that its publications align with the Department’s mandate and contribute to advancing its strategic priorities. | |
| 1.3 It is expected that NRCan provides sufficient direction and tools to enable an effective and efficient approach to the management of NRCan’s publishing activities. | |
| 1.4 It is expected that NRCan has adequate processes in place to ensure that its publishing activities are conducted in accordance with key provisions of applicable GoC and departmental policies. | |
| Audit Sub-Objective 2: To determine whether NRCan effectively manages its publishing activities by ensuring that its publications meet the needs of its intended audiences. |
2.1 It is expected that NRCan’s has processes in place to effectively disseminate its publications within the Department, and to stakeholders. |
| 2.2 It is expected that NRCan periodically revisits its approach to the dissemination and promotion of its publications to consider and adapt to the evolving environment. | |
| 2.3 It is expected that NRCan has measures in place to assess whether its publishing activities meet the needs of its intended audiences. |
Page details
- Date modified: