Audit of the SAP Functionality Project AU1307

TABLE OF CONTENTS

EXECUTIVE SUMMARY

BACKGROUND

Natural Resources Canada’s (NRCan) completed multiple reviews of the previous financial software and it was determined that the system was no longer meeting the department’s needs. A business case analysis was conducted and the recommendations stated that the financial system be replaced by SAP. Further, NRCan was to become a partner with Agriculture and Agri-Food Canada’s (AAFC) existing SAP system. In 2009, the Felix Project was launched to implement the new SAP system at NRCan, which has been operational since 2011. 

In 2011, a System Under Development (SUD) audit was performed on phase 1 of the SAP implementation. This audit identified that the project was on time and on budget, however other challenges remained, including both reporting and training issues.

This audit engagement was approved as part of NRCan’s 2012–2015 Risk-Based Audit Plan.

AUDIT OBJECTIVES

The overall purpose of this audit was to provide assurance that the SAP financial system is functioning to meet key business requirements. Specifically, the audit assessed:

  • Standard and custom reports generated by SAP meet business requirements and are easily accessible;
  • Processes and controls related to SAP security for current user roles are operating effectively;
  • SAP training currently offered to NRCan user meets established user requirements; and
  • NRCan is adequately monitoring and managing the Service Level Agreement and Operating Level Agreements.

SCOPE

The scope of the audit covered the SAP functionalities for the period of April 1st, 2011 to March 31st, 2013. Both the Service Level Agreement (SLA) and the associated Operating Level Agreements (OLAs) between NRCan and AAFC were included to the extent of how NRCan is managing and monitoring these agreements. 

The audit excluded areas that fall under the responsibility of AAFC as per the SLA dated February 7, 2011 which defines the service partnership with NRCan. This agreement identifies AAFC’s responsibilities for the provision of Integrated and Financial and Material System (IFMS) services, Project Systems (PS) and system support. The audit of both application controls and general computer controls were also excluded from the scope.

The audit approach was based on Treasury Board guidelines on internal auditing and standards defined by the Institute of Internal Auditors (IIA).

STRENGTHS

The audit identified a number of good practices currently in place at NRCan, with regards to the SAP system and related operations, including:

  • SAP security controls related to automated segregation of duties reporting and periodic monitoring of user roles;
  • Realization of initial training goals, and ongoing reporting on training statistics to management; and
  • SLA and OLA agreements between NRCan and AAFC outlining general roles and responsibilities.

AREAS FOR IMPROVEMENT

The results of our audit identified areas of improvement under each of the lines of inquiry initially identified for this audit:

  • Standard and custom reports currently generated by SAP do not meet the needs of many business requirements; therefore SAP information is exported to other tools, such as Microsoft Excel, to produce meaningful reports;
  • NRCan cannot determine if the current SAP training offered to its employees meets established user requirements as training effectiveness is not assessed; and
  • The Service Level Agreement (SLA) and Operating Level Agreements (OLA) do not include detailed information with respect to the delivery of key services. 

INTERNAL AUDIT CONCLUSION AND OPINION

The Audit Branch can provide reasonable assurance that NRCan is adequately monitoring and managing the Service Level Agreement and Operating Level Agreements and that SAP functionalities, as they relate to security for current users roles such as automated segregation of duties reporting and periodic monitoring are operating as intended. There are, however, gaps that exist in reporting and training.

In my opinion, there are opportunities for NRCan to improve both reporting and training activities, and to include additional key performance indicators in future SLA agreements associated with the SAP system at NRCan.

STATEMENT OF CONFORMANCE

In my professional judgement as Chief Audit Executive, the audit conforms with the Internal Auditing Standards for the Government of Canada, as supported by the results of the internal Quality Assurance and Improvement Program.

Christian Asselin, CPA, CA, CMA, CFE
Chief Audit Executive

INTRODUCTION

BACKGROUND

In 2002, Natural Resources Canada (NRCan) undertook the Integrated Management Information Project which concluded that the departmental financial management system, the Government Financial System (GFS), no longer met management information needs. NRCan initiated a re-examination of its systems renewal strategy in 2008 under the Financial Systems Renewal (FSR) initiative. The FSR initiative resulted in the delivery of a business case which recommended the replacement of the GFS with SAP, through a partnership with Agriculture and Agri-Food Canada (AAFC). In this partnership, AAFC, the Canadian Food Inspection Agency (CFIA) and NRCan share a single SAP system currently operated by AAFC, along with common business support processes. The resulting business case was approved in 2009 by the senior management of both NRCan and AAFC. NRCan launched the Felix Project to start the implementation of SAP on August 20th, 2009, which resulted in the formal system launch on April 4th, 2011. At the time of this audit, the SAP system had been in operation for two years and had gone through a full fiscal year cycle.

In the SAP business case, NRCan emphasized that the delivery of a new financial system on time and on budget was only one component of the renewal project. More importantly, NRCan was investing in a business process renewal initiative that, if functioning as intended, would result in eight (8) explicit business improvementsFootnote 1:

  1. Establishment of a departmental framework and supporting toolset for the planning and management of projects;
  2. Ability to support a controls-based audit of departmental financial statements;
  3. Ability to cost project initiatives;
  4. Ability to execute cost based billing for recoverable services (with supporting audit trail);
  5. Permit internal service charges in a manner that enables program managers to retain full-cost history of project /program delivery costs;
  6. Enabling service managers to retain full costs and billing history for key service units;
  7. Establishment of a corporate-wide planning/forecasting tool; and
  8. Real-time recording and reporting for financial commitments and budget availability.

A System Under Development (SUD) AuditFootnote 2 on Phase 1 of the project was conducted during the period of December 2010 to April 2011. The audit found that the Felix Project was completed on time and on budget as outlined in the original plan. However, several challenges were identified as part of the post-implementation, such as, improving system reporting, on-going training, and improvements to overall system capabilities. 

In part, these challenges were due to the limited scope of the project’s “Blueprint” that did not address areas such as post-implementation review, lessons learned or formal knowledge transfer given that the implementation team was no longer in place after the formal launch i.e. “Go Live” periodFootnote 3.

The effective functioning of the system’s post-implementation remains a departmental priority, given that the SAP system – as an Enterprise Resource Planning (ERP) tool – supports and impacts all the sectors and internal services within NRCan. 

AUDIT OBJECTIVE

The overall purpose of this audit was to provide assurance that the SAP financial system is functioning to meet key business requirements. Specifically, the audit assessed whether:

  • Standard and custom reports generated by SAP meet user requirements and are easily  accessible;
  • Processes and controls related to SAP security for current user roles are operating effectively;
  • SAP training currently offered to NRCan users meets established user requirements; and
  • NRCan is adequately monitoring and managing the Service Level Agreements and Operating Level Agreements.

DEPARTMENTAL RISKS

The 2012-15 Risk-Based Audit Plan, approved by the Deputy Minister, identified this audit as a priority for the department based on the following risk rationale:

  • Risk that the SAP reporting capability does not meet corporate and operational needs for decision-making purposes; and
  • Risk that system functionality is not leveraged to meet key business and operational needs.

In addition to these initial risks identified in the Risk-Based Audit Plan, the audit identified additional risks as follows:

  • Lack of mandatory standard reporting could promote the use of unauthorized, parallel and potentially erroneous methods for capturing and analyzing data;
  • User roles may be applied in a manner that adversely impacts on appropriate segregation of duties, creates delays in business workflow and causes added costs and inefficiencies to NRCan;
  • The training approach may reduce the ability of users to effectively adapt to changing job and business requirements;
  • The scope of information sharing may not be sufficient to support a fully integrated and flexible business environment;
  • The departmental management control framework (MCF) may not adequately address in-service needs, such as the management of the Service Level Agreement and Operating Level Agreements; and
  • NRCan may not have a suitable process for managing changes or an adequately integrated master plan in place to accommodate all change requests approved in accordance with the support management framework.

SCOPE AND METHODOLOGY

The scope of the audit covered SAP functionality for the period of April 1st, 2011 to March 31st, 2013. Both the Service Level Agreement (SLA) and the associated Operating Level Agreements (OLAs) between NRCan and AAFC were included to the extent that NRCan is managing and monitoring these agreements. 

The audit excluded areas that fall under the responsibility of AAFC as per the Service Level Agreement (SLA) dated February 7th, 2011 which defines the service partnership with NRCan.  This agreement identifies AAFC’s responsibilities for the provision of Integrated and Financial and Material System (IFMS) services, Project Systems (PS) and system support. The audit of both application controls and general computer controls were also excluded from the scope. 

The planning and conducting of the audit were based on professional standards to ensure that the audit’s findings and conclusions would be appropriate and consistent with the evidence collected. The internal audit process involved three main phases – planning, conducting and reporting – each of which was subject to a quality assurance peer review.

The audit approach addressed the stated objective against the audit criteria developed during the planning phase. Observations, assessments and conclusions were drawn using a detailed audit program to carry-out audit testing.

The audit approach was based on Treasury Board guidelines on internal auditing and standards defined by the Institute of Internal Auditors (IIA), and included:

  1. A review of relevant documentation, including the SLA and OLAs in place at the time of the audit, and other supporting documentation;
  2. Interviews with SAP users, focus groups, managers and select stakeholders;
  3. Walkthrough of the SAP training process and entry-level SAP e-learning;
  4. Check sheet procedures to collect and analyze SAP user’s interview results; and
  5. Audit testing based on judgemental sampling of reports and training tools produced by users.

AUDIT CRITERIA

The criteria were developed to guide the audit conduct and to form the basis for developing the audit testing activities and against which the overall audit conclusion and reporting is derived.

The specific audit criteria developed for this audit were:

  1. Standard and custom reports generated by SAP meet user requirements and are easily accessible.
  2. The processes and controls related to SAP security for current user roles are operating effectively.
  3. The SAP training currently offered to NRCan users meets established user requirements.
  4. NRCan is adequately managing and monitoring the Service Level Agreements and Operating Level Agreements.

See Appendix A for the sub-criteria related to each criteria.

FINDINGS AND RECOMMENDATIONS

REPORTING

Summary Finding

While employees are adapting to the use and application of SAP functionalities users continue to find solutions outside of SAP to satisfy their business reporting requirements due to a lack of mandatory standardized reporting.

Supporting Observations

SAP Implementation

SAP is an Enterprise Resource Planning (ERP) system. ERP systems make it easier to track the work-flow across various business units and reduce the operational costs of manual tracking and the possible duplication of data from using individual and disparate systemsFootnote 4. The use of ERP systems is a best practice for business operations.

In an effort to improve the business, including efficiency, reporting and decision making processes, NRCan implemented the SAP ERP system two (2) years ago. The implementation strategy was to adopt, rather than adapt (i.e. customize) the SAP applicationFootnote 5, as NRCan was and is sharing the application with AAFC. The result of this approach is that NRCan business processes are subject to modifications undertaken by the AAFC-led partnership.

One of the key business outcomes identified as part of the SAP implementation was “real-time recording and reporting for financial commitments and budget availability”Footnote 6 to improve decision support and better use of personnel. At the time of the audit, it would appear that this objective has not been realized. User satisfaction reporting does not exist, and there was no formal mechanism in place to respond to the reporting deficiencies identified by users.

SAP System Complexity

Managers find that the SAP system is time-consuming and complex to use. As a result, some managers have not utilized SAP to its full capacity, particularly for their forecasting needs and project management requirements. They view it as both difficult to navigate and lacking customized reporting.

Interviews with the users, combined with the analysis of the evidence gathered during the audit, confirmed that the frequent use of SAP increased the user’s understanding and proficiency of SAP reports. Some managers noted that because they found SAP reports to be difficult to use and as they do not produce them on a regular basis, administrative staff are often asked to generate reports. The audit team observed that some Sector Financial Advisors (SFAs) did not use SAP themselves but delegated the role to subordinate staff.  Interviews with senior management confirmed the delegation to other staff to produce some of the reports required to support decision-making and corporate reporting.

Use of Local Applications

SAP maintains a live database and does not freeze data at a given point-in-time, unless the user follows a complex process which can be difficult to master. To generate the periodic financial management reports in the formatting expected and required by management, users are exporting SAP reports to Excel workbooks for manipulation. As part of the conduct phase, the audit team obtained a sample of the Excel reports produced by various users that confirmed the widespread use of these reports as an alternative to SAP reports. Based on the audit team’s observations, the use of Excel reports as a means to supplement SAP reports did not impact the accuracy of the data in SAP.  However, the use of the Excel reports went beyond ad hoc financial reporting and is being used to meet business requirements. For example, Felix implemented the Project Systems (PS) Module of SAP to assist NRCan managers in managing their projects. However, project managers have created separate data streams for project monitoring in Excel workbooks, instead of using the PS Module output.  

Similarly, it was evident to the audit team that the Excel reports are also being used for forecasting. Though one of NRCan’s eight (8) explicit key business improvements included the establishment of a corporate-wide planning/forecasting toolFootnote 7, currently SAP does not provide a forecasting tool. To compensate, business units are inputting their plans, budgets, expenses, commitments and free balances for tracking and reporting (including feeding into the Branch forecast) in Excel tables.  They also use information from SAP to manually populate these Excel tables.

Where SAP reports are available, some users are not familiar with extracting the data or choose alternative data sources to manipulate such information for management reports. Such reporting rolled up to a corporate level may not fully represent SAP data. These concerns have been echoed by Business Process Owners at NRCan, who have identified forecasting and the Business Warehouse (BW) for reporting purposes as two key prioritiesFootnote 8 to help alleviate user challenges with reporting. Partners expressed their support of the BW and improved forecasting capabilities as a go-forward priorityFootnote 9 and the BW continues to be identified as a high priority item by AAFC-COE into 2013-14.Footnote 10

RISK AND IMPACT

Local applications, such as Excel workbooks, can be a significant risk if they are being used for departmental reporting and decision making. If these applications are considered as essential to meet business requirements, their inherent risks should be evaluated to ensure they comply with departmental policies, procedures and guidelines. Further, using local applications in conjunction with SAP causes inefficient resource usage, time allocation, duplication of effort, incompatibility across department, and increased operational burden on users.

RECOMMENDATION

  1. Corporate Management and Services Sector (CMSS) should provide mandatory standard reporting templates to meet identified business needs and corporate reporting requirements to ensure consistency and accuracy in reporting practices across all sectors.

MANAGEMENT RESPONSE, ACTION PLAN AND TIME FRAME

Management Agrees.

  • Executive Director Shared Services Office (ED SSO) and Director General Financial Management Branch (DG FMB) will meet with a Program Assistant Deputy Minister (ADM) and his/her supporting staff to discuss their issues with SAP reporting in order to identify appropriate improvement to be made to procedures, reports or training/awareness initiatives.

Timing: To be completed by December 2013

  • Director General Chief Information Officer Branch (DG CIOB), ED SSO, and DG FMB will also identify any other business needs and corporate reporting requirements to address issues with reporting in SAP or if additional training/awareness initiatives are required.
     
  • Business Process Owners (BPOs) will identify business requirements and how SAP reporting capabilities meet these requirements.

Timing: To be completed by January 2014

  • BPOs will consult as required with key stakeholders to determine if any business requirements have been missed and/or if SAP reporting capabilities are inadequate.

Timing: To be completed by March 2014

  • If any major gaps are identified between business requirements and SAP reporting capabilities, a formal plan will be developed to address these gaps. This plan will be developed in consultation with stakeholders and could include the use of solutions within or outside SAP.

Timing: To be completed by March 2014

SECURITY AND ACCESS

Summary Finding

In order to achieve appropriate segregation of duties, a procedure exists to monitor and validate the assignment of SAP roles to users.

Supporting Observations

Security and Role Management

Security in SAP is a shared responsibility; NRCan requests access and AAFC (through the Centre of Excellence (COE)) grants it. In order to obtain user access, training must be approved by the responsible manager and the business process owner at NRCan. Most of the technical aspects of security are system controlled and fall under the responsibility of AAFC (COE). To this extent, NRCan is a “client” and controls only the front end of the process. At the time of this audit, there were no formal procedures in place, including any required documentation and approvals.

Access rights within SAP are based on a set of standard roles. Operational duties may require one or more standard SAP roles to provide a profile with sufficient data access and reporting capabilities. The SAP system can assess role combinations, detect any conflicts related to the segregation of duties, and produce a report whenever this situation occurs.

With this in mind, roles that are requested by managers are always examined by NRCan for potential conflicts, even before a training request is sent to the AAFC-COE. If any questions related to role compatibility are raised, the request is sent back to the manager. If a manager can demonstrate that a combination of roles is necessary, the Business Process Owner Committee (BPOC) can approve this exception.

Segregation of Duties

The “Segregation of Duties” report outlining incompatible roles is available to the Shared Services Office (SSO) of NRCan, which is responsible for the SAP security administration.  During the audit, it was noted that the existence of this report was not common knowledge to anyone outside the SSO. However, whenever there was a potential for inappropriate segregation of duties, the SSO received a system-generated report, and shared this information with the appropriate manager. This manager would be asked to analyze and explain the conflicting duties to the satisfaction of the members of the BPOC. If the BPOC agreed, this exception was allowed and management accepted, on a case by case basis, the risk of conflicting segregation of duties. At the time of the audit, SSO was also carrying out – at entity level – the monitoring of user access by sending periodically letters to all managers to review and confirm the access rights of their employees.      

In addition, NRCan has implemented an “Exit Checklist” to be used whenever an employee leaves the department. The audit team also found that access lockouts were triggered automatically by the system should an account be inactive for 90 days, or manually if SSO received notification that a position has been vacated. However, there were no processes in place to immediately cease SAP role access when an employee changed their job position in the department. As a result, there is currently a possibility that employees who change positions frequently may accumulate roles that exceed their actual job requirements for SAP access. It was noted during this audit that NRCan recognized these issues and management was taking action.

RISK AND IMPACT

SAP user roles are controlled at the system level by the Corporate Management and Services Sector – Shared Services Office (CMSS-SSO), and a formal approval is required for any exceptions. Most employees have a standard role in the system and role management is generally transparent to managers and staff. Role management ensures that employees leaving the department have their access revoked. As discussed in the section above, new controls are being planned for employees changing jobs within the department.

RECOMMENDATION

No recommendation identified.

TRAINING

Summary Finding

NRCan has achieved the initial goals of providing SAP training to staff and managers. However, after two (2) years of operations, training needs have changed. There is no formal strategy in place to address the changing needs for training among NRCan’s SAP users. As a result, NRCan staff are not using the SAP system to its full capacity to reduce the day to day effort of NRCan personnel and to improve decision making.

Supporting Observations

The terms of the SLA state that the AAFC-COE provides core SAP training with no direct billing. It is well-understood that core SAP training through the AAFC-COE is a pre-requisite for SAP access. The OLA outlines core training for the SAP solution, to include delivery by instructor, on-line, video-conference and coaching (one on one). The SLA commits the AAFC-COE to perform periodic review of users. 

Changing Training Needs

At the time of the audit, it had been more than two (2) years since SAP’s introduction, the learning curve for SAP users appears to be flattening out. Most managers and staff that were interviewed acknowledged that both the initial training, and the subsequent, more formal training through the AAFC-COE, have been generally effective in helping users meet their business requirements.

Though the initial training has been largely completed, a new need for specialized training has arisen and has not been addressed. Expert users, for instance, have very limited targeted training support. The Felix Blueprint recognized these “power users” as an Expected Change Management Impact – particularly in the area of reporting: “This will be a new process that will require a training effort for super and power users”Footnote 11. While the “power user” concept was not formally implemented at NRCan, there are clearly staff who are informally considered “power users”. The only training course noted in the SAP Course Catalogue was an optional 3-day AAFC-COE course (SAP013Footnote 12) for the SAP Salary Forecasting Tool (SFT) for Power Users.

As a result, managers and users are left on their own to address the development of additional SAP skills. For example, one sector contracted additional training from an external supplier to help meet their training needs. This sector’s management reported that the employees viewed this type of training as effective and beneficial. 

Alternative Means of Information Sharing

Information sharing regarding SAP operations and SAP reports is usually done via AAFC-COE bulletins to SAP users or Financial Management Bulletins through the SSOFootnote 13. Users were generally aware of these bulletins; however, 82% of users interviewed relied upon other means of sharing information regarding SAP functionalities and reports. These included peer-to-peer coaching via emails, informal “how-to” documentation and referrals to business process SAP power users. Users perceived these alternatives as effective within their teams, but communications rarely went beyond their team and the administrative staff community.

Other examples of information sharing included several “How To” instructions reviewed by the audit team that were produced by users to explain how to download SAP data into an Excel worksheet to produce reports. These instructions confirmed that there was a need to learn how to download and manipulate SAP data in an alternative application, which supports the previous finding that the SAP system does not currently achieve users desired operational reporting needs.

Limited Training Statistics

Quarterly reports presented to the SAP Partnership Management Committee (SPMC) include the number of registrants, participants, no-shows and cancellations, as well as a summary of evaluations received. The audit team found that the AAFC-COE presented to both, the ADM SAP Partnership Steering Committee and the SAP Partnership Management Committee, efficiency-based statistics for each departmentFootnote 14 (i.e. number of staff trained per period, resources utilized to deliver training, etc.). However, at the time of the audit, the AAFC-COE’s operational updates did not have any qualitative assessments, such as course evaluation summaries, to provide feedback on the effectiveness of the training.

SPMC minutes did not indicate if NRCan followed-up to obtain qualitative assessments, nor did the audit team find a request from NRCan management to the AAFC-COE for feedback on training effectiveness. Such reporting could have assessed the user satisfaction, including applicability to job requirements, use of time and timeliness of offering. For example, an informative statistic for management to consider to address the effectiveness of training is the training delivered by staff level. An audit analysis of the training taken by NRCan users indicated that managers accounted for 21% percentage of the SAP training instances and non-management staff accounted for 79%.

During the interviews, users expressed dissatisfaction with the core SAP training, noting that it did not provide sufficient depth for users and limited their ability to effectively and efficiently use the SAP reporting function.  Based on the results of the audit work, there is evidence that the quality of the SAP training was largely dependent on the trainer’s expertise, the classes were often too long (e.g. 3 days for Project Systems Training) and the course content was too generic.  Overall, the SAP training was viewed as being carried out by instructors who knew the SAP modules, but had little knowledge of NRCan’s operations.  One manager noted that there were no tests at the end of the courses to validate a user’s ability to apply the skills learned. At the time of the audit, the core SAP classroom courses were pre-scheduled and if online training was not available, new employees could not take their training in a timely manner, thereby affecting their access and their ability to meet operational needs.

RISK AND IMPACT

Recognizing that training needs are changing is a fundamental part of organizational change management. Without the recognition that training needs are evolving, there is a risk that users will be inadequately prepared to respond to changing business requirements, and users will gain less and less benefit from the current set of core SAP courses.

RECOMMENDATION

  1. Corporate Management and Services Sector (CMSS) should work with partners and/or consider internal Natural Resources Canada (NRCan) training programs to address training on business and system procedures which is beyond the basic training programs currently offered. Regular follow up reporting should be communicated to senior management, including mechanisms and metrics to assess the effectiveness of the training.

MANAGEMENT RESPONSE, ACTION PLAN AND TIME FRAME

Management Agrees.

SAP training is part of the regular agenda of the SAP Governance structure (Assistant Deputy Minister (ADM) SAP Partnership Steering Committee, Director General (DG) SAP Partnership Management Committee, NRCan Director-level Business Process Owner Committee and SAP Partners/NRCan working level meetings). In FY 2014, NRCan has identified E-Learning (a component of Training) as one of its top five SAP priorities, to be addressed through the SAP Partnership.

This governance structure is already in place.

Director, Finance and Procurement has recommended to the Centre of Excellence (COE) that surveys completed by users at the end of each SAP course be modified to meet follow up reporting needs to senior management

Timing: To be completed by March 2014

Business training needs are the responsibility of individual departments and most recently a new reporting course for managers and for administrative employees was developed by the Business Training Unit of CMSS. In addition, the Unit is developing a Procure to Pay course to be offered in October 2013. The unit will also review other key end-to-end business processes to see if other business process courses are required.

Responsible position: Corporate Management and Services Sector Associate Executive Director Shared Services Office

Timing: To be completed by June 2014

CMSS will develop surveys to be completed by users at the end of each business process course  that  meet follow up reporting needs of senior management.

Responsible position: Corporate Management and Services Sector Associate Executive Director Shared Services Office

Timing: To be completed by June 2014

SLA MANAGEMENT 

Summary Finding

SLA and OLA agreements between NRCan and AAFC are established and documented, outlining general responsibilities and roles. In negotiating the SLA/OLA agreements with AAFC NRCan should be more specific in detailing their role in the various areas and should clearly state the requirements for monitoring the delivery of key services with key performance indicators.

Supporting Observations

The OLA appendices to the SLA provide process documentation that adequately and sufficiently covers the subject areas. In general, NRCan makes use of pre-existing AAFC processes and once a Change Management, Release Management or Training request is made, AAFC manages the process to move forward and implement the request. Overall, the review of the documentation found no specific statement of performance.

Monitoring and Reporting

The SLA and OLA standards are generic, and NRCan managers are unclear as to how and where the service standards originated. The COE has quarterly reporting requirements on quantitative measures, such as training volume, Help Desk calls and response time, fiscal year priorities for development, change requests and NRCan resources at the COE. The audit team found that the reporting is only specific when service standards are explicit. For example, the SLA indicates that AAFC will acknowledge receipts to clients 90% of the time within the values noted in the agreement.

The following quote is evidence that AAFC has met this service standard for NRCan 95% of the time: “Successful Acknowledgment by IT Centre as per the Service Level Standard: AAFC=85%, CFIA=65%, NRCan 95%”Footnote 15. Within NRCan some informal feedback has been captured (e.g. Help Desk monitoring, course evaluations) with occasional activity reporting made to the BPOC and to the Partnership Committees.

However, based on the audit work, effectiveness reporting, such as overall system performance and assessment of user satisfaction, has not been requested by the SPMC and nothing has been formally documented regarding SLA management, monitoring and reporting at the departmental level.

Agreement Renewal

In the discussions to update the current SLA which expired on March 31, 2013, SPMC committee minutes demonstrate that there is an increase in engagement by NRCan to seek additional information for the SLA and internal management, along with partnership requests to include an audit clause for partner audits and a longer term SLA (5 years) including variable costs for monitoringFootnote 16. BPOC members that were interviewed acknowledged that there were still challenges in identifying suitable measures of service delivery.

In addition, the audit team found that there were areas of responsibility assigned to AAFC but NRCan’s responsibilities were unclear. For example, the disaster recovery of the data centre being used to run SAP is the responsibility of AAFC.  However, it would be expected that NRCan also has a role to play should such an event occur. AAFC’s expectations of NRCan’s role should also be clearly identified in any new SLA and the two roles should be in alignment.

RISK AND IMPACT

As a relatively new member of the SPMC, NRCan has been in a learning phase, and putting in place effective mechanisms to monitor and manage service levels has not been a priority until now. As a result, there is a lack of information on which to base the monitoring of SLA service standards and the negotiation of changes to service delivery. There is a risk that future SLAs may not adequately address NRCan’s in-service needs.

RECOMMENDATION

  1. Corporate Management Services Sector (CMSS), on behalf of Natural Resources Canada (NRCan), should take advantage of the current Service Level Agreement (SLA)/Operating Level Agreement (OLA) agreement renewal process to further clarify responsibilities of NRCan and Agriculture and Agri-Food Canada (AAFC). The revised agreement should include a new mechanism for monitoring the delivery of key services with key performance indicators.
     
  2. CMSS should include SAP Issues as a standing item at the Shared Services Office (SSO) Client Advisory Committee meetings to ensure that user needs are identified and actioned as required.

MANAGEMENT RESPONSE, ACTION PLAN AND TIME FRAME

  1. Management Agrees

During the Service Agreements (SAs) renewal process, the Director General Financial Management Branch, Chief Information Officer and Executive Director SSO are responsible for identifying the mechanisms for monitoring the delivery of key services with key performance indicators.

Timing: SLA and Service Agreements (SAs) to be completed by December 31, 2013

  1. Management Agrees. 

CMSS will begin including SAP issues as a regular standing item at the SSO Client Advisory Committee.

Responsible Position: Executive Director, Corporate Management Services Sector  

Timing: To be completed by September 2013

APPENDIX A – AUDIT CRITERIA

Actual performance was assessed against the audit criteria resulting in either a positive finding or the identification of an area of improvement.

Audit Criteria Used to Conduct the Audit
Audit Sub-Objectives Audit Criteria
Sub-Objective 1: Standard and custom reports generated by SAP meet user requirements and are easily accessible. 1.1 SAP reporting information is presented and distributed in a form and timeframe that supports reporting requirements and enables users to carry out their responsibilities. (OCGFootnote 17: ST-18 – Stewardship & Reporting; CobITFootnote 18: DS11 – Manage Data)

1.2 Feedback from users and other stakeholders is used to ensure SAP reports are relevant and aligned with user needs; and, to identify opportunities for enhancing reports. (OCG: CFS-2 – Client Satisfaction; CobIT: DS8 – Delivery and Support)
Sub-Objective 2: The processes and controls related to SAP security for current user roles are operating effectively. 2.2 NRCan has implemented processes and controls related to SAP user security to help safeguard access and ensure that data is appropriately used and maintained, and users can effectively carry out their responsibilities. (OCG: ST-23 – Stewardship & General Information Technology; CobIT: DS5 – Delivery and Support)
Sub-Objective 3: The SAP training currently offered to NRCan users meets established user requirements. 3.1 NRCan provides employees with the necessary SAP training, tools, resources and information to support the discharge of their responsibilities. (OCG: PPL-4 – People); CobIT: DS7 – Delivery and Support)

3.2 An information sharing/communications process exists to support the efficient and targeted dissemination of relevant and reliable SAP information to those that need it. (OCG: PPL-4 – People); CobIT: DS7 – Delivery and Support)
Sub-Objective 4: NRCan is adequately managing and monitoring the Service Level Agreements and Operating Level Agreements. 4.1 Management has put in place an effective mechanism to monitor and manage the attainment of service level commitments specified in the SLA and the associated OLAs. (OCG: RP-3 – Results and Performance); CobIT: ME2 – Monitor and Evaluate)