System Under Development Audit of the Felix Project (AU1105)

Reports 2011


Table of Contents


EXECUTIVE SUMMARY

Introduction

This report provides a synopsis of the results of an internal audit undertaken during the 2011 implementation of a new integrated financial and materiel management system for Natural Resources Canada (NRCan).  The system implementation was based on an agreed service partnership between NRCan and Agriculture and Agri-Food Canada (AAFC) for the provision, by AAFC, of Integrated Financial and Materiel System (IFMS) services, Project Systems services and system support, using the commercial software Systems Applications and Products in Data Processing (SAP).

This synopsis report presents key issues identified and communicated to senior management during the audit as a result of a specific audit approach and methodology described as a System Under Development (SUD) Audit. Going forward, a number of SAP implementation challenges remain that will require ongoing due diligence through the post-implementation period.

The audit was conducted during the period from December 9, 2010 to April 4, 2011.

Background

In 2002, NRCan undertook the Integrated Management Information Project which concluded that the departmental financial management system, the Government Financial System (GFS), no longer met management information needs.  NRCan initiated a re-examination of its systems renewal strategy in 2008 under the Financial Systems Renewal (FSR) initiative. The FSR initiative resulted in the delivery of a business case which recommended the replacement of the GFS with SAP, through a partnership with AAFC.  In this partnership, the departments share a single SAP system, along with common business support processes. The resulting business case was approved by the senior management of both departments in 2009.  NRCan launched the Felix Project to implement SAP on August 20, 2009.  A memorandum of understanding was signed by NRCan and AAFC.Footnote 1

The main functionality of the new integrated system includes finance, materiel management / procurement and project management.  The business approach used was that NRCan adopted existing AAFC business processes for finance and materiel management, and would develop the project management functionality needed to meet NRCan requirements.  NRCan had a system launch or ‘Go-live’ on April 4, 2011, at an estimated implementation cost of $14,800,000.  As presented in the approved Felix Project Charter, the total project cost over four years, including project contingency, was estimated at $14,788,000. 

The audit was conducted in accordance with Treasury Board Policy on Internal Audit and the prescribed standards of the Institute of Internal Auditors (IIA). These standards require that the audit be planned and performed in such a way as to obtain reasonable assurance that audit objectives were achieved. Consequently, the audit included various tests and review of documents, as considered necessary, to provide such assurance. These tests included, but were not limited to headquarters, document reviews, observation, examination of project team planning documents, budgets and assessment of compliance to applicable authorities and references (See Appendix C References).

Internal Audit Conclusion

The audit objective was to assess the Felix Project with respect to the completion of project deliverables against original plans (i.e., costs, functionality and time) and provide feedback to the department on issues and concerns that could affect the success of project delivery.

Overall, the Felix Project was completed as outlined in the official project plan, on time and within budget. The audit found that management processes were in place to adequately and effectively manage the risks associated with project milestones, costs and scope. The launch for the application occurred on April 4, 2011 according to schedule and within its implementation budget of $ 7,593,800 for 2010-2011.  The total direct project cost estimated for the current year (i.e., 2011-2012), the year in which the enterprise application first operates – is $1,630,000, including a project contingency of $207,000. 

This positive accomplishment must be placed in the context that a number of SAP implementation challenges remain such as implementing new financial management and reporting capabilities that were originally designated as out of scope for the project. Given the extensive amount of ongoing development with the AAFC-NRCan amalgamated SAP Centre of Excellence, on-going training, hardware and software adjustments, business transformation, application and business stability efforts and related communications required this year, it appears the $1,630,000 budgeted for 2011-2012 for Felix Project implementation may only support a portion of the total cost to the department. If ongoing SAP-related development costs prove to be significant, this could represent an unanticipated level of effort that the Department will have to address into the foreseeable future with a well-crafted governance model, management control framework and resource planning methodology. 

In the context of continuing resource constraints, these challenges will require ongoing due diligence through the post-implementation period.

Audit Objective

To assess the status of the Felix Project with respect to the completion of project deliverables against original plans (i.e., costs, functionality and time) and provide feedback to the Department on issues and concerns that could affect the success of project delivery.

Methodology and Approach

The System Under Development (SUD) audit methodology requires applying a risk-based approach to determine which project areas are critical to the success or failure of a project.  On the basis of various issues such as project governance, planning, resourcing, staffing, risk management and project progress, the audit team selected topics for examination where critical challenges to the project were anticipated. This resulted in audit resources being focused on higher risk areas. (The Lines of Enquiry used for the audit are provided in Appendix A.)

During SUD audits, audit teams maintain an on-going presence in the system development project. The audit was conducted in phases that corresponded to key project development phases or milestones. During this audit the audit team was co-located and worked alongside the project development team(s) to provide timely advice and guidance as the project progressed. In addition, the audit team leader or manager sat as an observer at the weekly meeting of the development team leaders, and the audit director or Chief Audit Executive sat as an observer at the senior project steering committee. In this way, the SUD audit provided ‘real time’ feedback to the project team and to senior management on issues and concerns that could affect the success of the project. The audit team reported to management as the project progressed prior to the “Go-live” date on those items requiring senior management response and action in the near or short-term.

During this audit the audit team reviewed information and documents as they were being produced by the development team, and used interviews to understand and confirm the project team strategy, plans and tasks before they were implemented. As part of the audit, the audit team interviewed project team members, and other key stakeholders (See Appendix B for a list of Audit Participants).

The audit team review included (but was not limited to):

  • Key deliverables and planning documents;
  • Project risk registers;
  • Project testing documentation;
  • System and interface data flow diagrams;
  • System security documentation;
  • Project Quality Assurance documentation;
  • Various end-user sign-off documents;
  • Detailed master schedules; and
  • Executive Dashboard reporting.

Throughout this report various organizational acronyms are used. For a complete description of these acronyms see Appendix D.

Overview of Lines of Enquiry and Analysis

Project Reporting to Management

Synopsis:  Felix Project reporting to management was comprehensive and timely, allowing departmental management to receive a clear picture of project  progress, issues and capabilities leading up to the April 4, 2011 ‘Go-live’ of SAP.

Throughout project implementation, senior management of the Department received comprehensive and accurate project information.  The project dashboard and management report for the Assistant Deputy Minister was updated on a weekly basis.  An executive-level dashboard was updated each month for the Felix Project Steering Committee. A progress report identified current project activities by category and reported the status or results as required.  Project progress and any adjustments to plans were tracked against the project schedule and were reported to management.  A project risk register was reviewed each week and was updated by the Felix Project Office Manager in consultation with the  application development Team Leaders and the status of key project risks were included in management reports.  Important project decisions and related actions were tracked and reported in an Action Register and in Status Reports.  Any unresolved issue generated an Action Register item which was reviewed weekly and included in a monthly report to the Steering Committee.

All project budget and expense information, both in detailed and summary format, were included in management reporting.  The budget was based on realistic end-dates for milestones; not hopeful forecasts.  Year-to-date actual and accrual figures, and forecast-to-year-end figures were reported.  The Project Office monitored contracts that needed to be extended based on additional work identified as the project progressed. Project accomplishments and upcoming activities and deadlines were reported.  At the time of the audit, a complete record of all Felix Project management reporting was maintained by the Project Office.  Business change management plans, issues and progress were also reported to senior management, as was the Sector Readiness scorecard for the April 4, 2011 ‘Go-live’.  The AAFC-NRCan Partnership Model Update was also presented to senior management.

Conclusion

The audit concluded that the Felix Project reported in a comprehensive and timely manner to senior management. This reporting addressed all the key aspects of project activities and included budget and expense items; schedule and progress; risk and risk mitigation; issue and issue management; decisions and actions; contracts and extensions; accomplishments and upcoming activities; business change management plans, issues and challenges; Sector readiness scorecards; and AAFC-NRCan Partnership Model update information.  

General security Processes and System Security Requirements

Synopsis:  General security processes and system security requirements were sufficiently addressed, with the exception of a timely process for deactivating the accounts of employees who had either left the Department or who had changed job duties.  At the time of the audit, a comprehensive Disaster Recovery Plan was not in place for SAP; however, interim measures for data recovery were taken by senior management. In addition, with regard to security incident monitoring, the audit team reviewed the Service Level Agreement (SLA) for the ‘Converged Network Service’ between PWGSC and AAFC and noted that the SLA did not provide sufficient clarity as to the monitoring and reporting processes in place to apprise NRCan of security incidents occurring on the interdepartmental telecommunications link. These issues were addressed by Management during the audit.

System Development and Implementation Security

In accordance with the NRCan Directive on IT System Development and the NRCan Privacy Impact Assessment Policy, the audit team examined the completion of three security-related documents required for implementation of new systems:  Statement of Sensitivities (SOS), Threat and Risk Assessment (TRA), and Privacy Impact Assessment (PIA) or Preliminary-Privacy Impact Assessment (P-PIA). 

In accordance with the NRCan Directive on IT System Certification and Accreditation (C&A), the audit team reviewed compliance with the requirements intended to provide reasonable assurance that new systems are secure prior to implementation into the production environment.  The Certification and Accreditation process, however, could not be completed by the Go-live date of April 4, 2011, as the action plan on recommendations from the Threat and Risk Assessment for SAP implementation had not been completed.  (Note: The TRA was finalized on June 1, 2011.) As a result, Certification and Accreditation for SAP was not completed.  Consequently, the departmental IT Security Coordinator provided the SAP system with an Interim Authority to Operate (IAO) valid until February 1, 2012.  The Audit recommended that a follow-up of the Certification and Accreditation process be conducted on February 1, 2012.

Segregation of Duties within the System (SAP Authorizations and Job Roles)

The audit team examined evidence that structured and documented processes were followed in order to design and configure SAP user roles compliant with NRCan business processes requirements, that proper segregation of duties was considered, and that the NRCan business owners responsible for the various SAP modules were consulted during this process.  The audit team also examined evidence that a process was adequately designed to request, approve and provide users with access to SAP.   

Network and Infrastructure Readiness and Security

The audit team examined evidence that departmental network and infrastructure risks, remediation, and monitoring and reporting had been identified, assessed, planned, resourced and tested in support of the implementation of SAP.

At the time of the audit, a comprehensive Disaster Recovery Plan was not in place for SAP; however, interim measures for data recovery were taken by senior management.  In addition, with regard to security incident monitoring, the audit team reviewed the Service Level Agreement (SLA) for the ‘Converged Network Service’ between PWGSC and AAFC and noted that the SLA did not provide sufficient clarity as to the monitoring and reporting processes in place to apprise NRCan of security incidents occurring on the interdepartmental telecommunications link.

Audit Branch informed management regarding these issues during the audit and a remediation plan was prepared to address these findings.
 
Regarding system performance and stress testing, the audit team noted the Felix Risk Register identified ‘Infrastructure capacity’ as a risk item, and related contingency measures were to do stress testing and monitoring during the test cycle.  Due to time and resource limitations, however, stress testing for SAP was not conducted during the test cycle.  Performance and stress testing are important steps in the development of a system to determine how well the system performs under a particular workload as well as to determine the application's robustness in terms of extreme load, which helps application administrators evaluate whether the application will perform sufficiently if the current load goes well above the expected maximum.  Following a documented stress strategy and plan, one day of stress and performance testing was conducted on March 25, 2011, 5 working days before SAP ‘Go-live’.   

Conclusion

The audit concluded that controls related to SAP user access management and general security processes were sufficient at the time of the audit with the exception of a process for deactivating SAP user accounts of employees who have left the Department or who have changed job duties within NRCan. The audit recommended the implementation of a process to notify the SAP Security Team when an SAP user leaves the Department or when an SAP user changes job positions within NRCan. Senior Management responded that a process would be in place by March 31, 2011.

Training

Synopsis:  Designated Wave 1 training, for initial SAP users to be ready for the April 2011 ‘Go-live’ of SAP, was conducted as scheduled and, in general, with trainee satisfaction.  Trainees self-evaluated their state of readiness to use SAP as Moderate, which is common with first-time users of SAP.  Of concern, however, was the 13% ‘no show’ of registered trainees at their training sessions.

The audit review of a random sample of Felix Training Evaluation forms, from the period February-March 2011, indicated that while SAP trainees were generally satisfied with training course content and structure, and with instructors, their ability to apply new operational concepts and SAP system skills was self-rated at Moderate  (4.6 on a scale of 1-7).  Of concern was that SAP trainees who remain uncertain about operational and system-use proficiency could translate into a higher frequency of telephone calls and e-mails to the SAP Help Desk during the post-implementation period.

Also, approximately 13% of Wave 1 (February - March 2011) registered trainees did not show up for training. The no shows’ appear to have been concentrated during the ‘March break’ in both Ontario and Quebec. The lack of SAP training for key employees may have some impact on SAP acceptance and support from the user community during the new fiscal year.

The Audit Branch was advised that senior management took action to ensure that those employees who had failed to report for their registered training courses, were identified and received the requisite training at a later date. As well, system access was not provided to employees until they had completed their training.

Conclusion

The audit concluded that SAP training across the Department was delivered successfully and on schedule, with the most significant concern being the registered trainee ‘no show’ rate on the initial training prior to “Go-live”. (Note: The audit field work ended April 4 and so the focus of the review dealt with training pre-Go-live.)

Data Conversion and System Interfaces

Synopsis:  Data Conversion involves the process by which data was obtained from the legacy financial system and loaded into the SAP application to ensure data completeness and integrity, whether through automated or manual conversions or master data loads. This area had a number of challenges and issues due to time constraints, however the data conversion was sufficiently complete as of April 4, 2011 to be considered satisfactory. Another dynamic area of development involved the establishment and testing of system interfaces into and out of the new application. All interfaces were subject to a documented process to describe their technical requirements, prepare cost estimates (for their development), and testing prior to transfer into the production environment.

Data Conversion

A review of various elements of data conversion activity was performed. The review included the GL organizational structure and Chart of Accounts, Master Data loads and Closing / Opening Balances.

Data conversion involves the process by which data is obtained from the legacy system and loaded into SAP-Felix with a view to ensuring data completeness and integrity, whether through automated or manual conversions or simple master data loads.

The audit team conducted interviews with individuals in the Financial Management Branch (FMB) of NRCan and reviewed documentation, on-going conversion status reporting and executive dashboards as well as various other control documents to identify potential issues.

  • General Ledger and Chart of Accounts
    The chart of accounts provides a framework to identify, aggregate, and report financial transactions for the Department. In government SAP implementations there are three charts of accounts for internal and external reporting purposes. The General Ledger (GL) is the complete record of all business transactions in NRCan.

    The three charts of accounts are:
    1. Chart of Accounts – Agriculture and Agri-Food Canada (CAGR)
    2. Economic Object Financial Coding (ECON)
    3. Financial Reporting Account (FRA)

    The operational chart of accounts at NRCan prior to conversion contained approximately 1,100 accounts. It was recognized that the Department would need some additional revenue, liability and expense accounts unique to its requirements. At the April 4 ‘Go-live’ date, a small number of these accounts (15) had not yet been set up. 

    Special Purpose Accounts (SPAs) - unlike lapsing authorities - roll-over every year.  At the time of the audit, work to find discrepancies of about $220,000 at the FRA-level between liability balances and cash balances was underway. This was required before scheduled conversion could occur.

  • Master Data
    At the time of our audit, Master data was being loaded to SAP-Felix according to scheduled priorities.

    Vendor Master data, involving some 21,000 vendor accounts, was successfully converted and accepted by the various areas in the department responsible for the processing and handling of these accounts.

    Customer Master Data conversion, comprising approximately 8,700 entities to which NRCAN sells and invoices goods and/or services and receives payments, continued as scheduled. The target date for completion was May 2011.

    Asset data processes transfer master data information and asset values (opening balances) as at April 1, 2011. Asset data must be prepared and cleansed prior to populating the asset sub-ledger. As of April 19, 2011 the area in the Department responsible for the processing and handling of the departmental asset inventory had reported that the inventory was approximately 95% complete, with outstanding items at the senior manager level for action. As a result there were some delays in reconciling inventory and finalizing the Asset Accounting (AA) and Plant Maintenance (PM) modules of SAP. Capital Asset conversion was scheduled for mid-May, with a GL adjustment anticipated in GFS P12-2 or P12-3.  NRCan has approximately 55,000 assets (includes 6,000 capital assets and 10,000 inactive assets). Capital Assets were a priority for inclusion in the AA module; the remaining non-capital (active) inventory were to be populated into the Plant Maintenance module.

All GL accounts that require support via a subsidiary ledger with reported balances in the final GFS Central Financial Management Reporting System Trial Balances (including A/P, A/R, assets, etc) must first be reconciled then converted to opening balances for the fiscal year 2011-2012 in SAP.

Long-standing issues related to FRA mapping changes in 2002 were attributed to a variance of approximately $10 million in the Capital/Fixed Assets sub-ledger balance as at mid-April. This was addressed and resolved prior to conversion in mid-May 2011. Work was undertaken to eliminate a discrepancy of approximately $500,000 in the non-capital asset sub-ledger.

Opening balances are normally available by the close of Period 12-2 (May); however, it was expected that data cleansing and conversion requirements would delay conversion. An extension to the completion of information for the Receiver General was requested by NRCan and acknowledged by the Receiver General in March 2011.

  • Closing and Opening Balances

The risks foreseen with data conversion were identified in project team planning prior to implementation. The SAP implementation was completed on April 4, 2011 as scheduled.

Conclusion

While data cleansing and balance reconciliations proved to be challenging due to implementation time pressures, data conversion progressed to the point where the completeness and integrity of the data loaded into SAP was satisfactory.

System Interfaces

As of March 30, 2011 the project had identified 6 inbound and 24 outbound interfaces. The Business Blueprint also identified 4 other files that would be provided to the Office of the Auditor General.

As of April 4, 2011 there were 18 interfaces in production and 12 additional interfaces were planned to be put into production between April 18 and May 15, 2011.

The testing of interfaces followed a documented process which included two key aspects:

  • Firstly, a procedures document was prepared at the start-up of the project which outlined roles and responsibilities for team members and data owner.
  • Secondly, as testing was conducted, a centrally accessible log to track testing and results (i.e. Pass/Fail) and sign-off by business owners was maintained. 
Conclusion

All interfaces were subject to a documented process to describe their technical requirements, prepare cost estimates and to move into a production environment. After review, the audit conclusion was that the processes in place for managing system interfaces, from identification, cost estimate preparation, and testing prior to entry into the production environment were adequate.  

Project Implementation and Roll-out strategy

Synopsis:  The implementation and roll-out were conducted as planned, and in accordance with the Felix Blueprint.   A number of anticipated and common SAP implementation challenges remain that will require ongoing due diligence through the post-implementation period and for the balance of the current fiscal year.

The project implementation and roll-out strategies were comprehensive, integrated with other corporate and partner strategies, and were supported at the senior management level. The audit noted that the roll-out strategy was consistent with the Felix Blueprint and was sufficiently integrated with plans and initiatives of branches, regions and partner organizations. The roll-out strategy included risk management and issue escalation, as needed.

The implementation and roll-out plans addressed IT resources, including hardware, software, facilities and materials.  The plans identified personnel requirements, including staffing and training needs; risks and issues, the impact of implementation; performance monitoring and configuration management plans (e.g., roll-out and upgrade of versions and supporting software and hardware were also addressed).  Also, the budgeting and costs for Felix implementation and roll-out were effectively documented. Plans were formally documented and vetted at an appropriate level.

The ongoing status of Felix implementation and roll-out was fully reported to senior management through the use of the ADM Sponsor Weekly Dashboard and at Felix Steering Committee meetings.  The roll-out was to be run in parallel with the legacy systems.  From April 4 to June 30, 2011, legacy financial systems were run to process transactions in order to close-out the 2010–2011 fiscal year (e.g., to pay prior year invoices) and to establish year-end closing balances.  New fiscal year transactions (i.e., 2011-2012) were processed in the new SAP application. 

Implementation Phases

To implement SAP effectively, implementation and roll-out phases were used.   Planned phases included detailed criteria for final implementation, application set-up in the various development, test, quality assurance and production environments, systems security set-up for live use (i.e., development, testing and training), and hardware, software and communications availability.  Site certification, controlled data conversion, formal cut-over planning to the new system, and user and project team sign-off were all planned and applied.

Post-Implementation Challenges

Following ‘Go-live’ a number of implementation and roll-out challenges remain.  These challenges include user support, technology management and support, information management, and records management.

The audit team briefed senior management on specific areas where ongoing due diligence will be required and a remediation plan was prepared to address these challenges.

Conclusion

Project Felix Implementation and Roll-out Strategies were comprehensive, integrated with other corporate and partner strategies, and were fully reported and supported at the senior management level.  The roll-out was run in parallel with the legacy systems; the former processing new-year transactions; the latter completing year-end processing and establishing year-end closing balances.  Going forward, a number of anticipated and common SAP implementation challenges remain that will require ongoing due diligence through the post-implementation period and for the balance of the fiscal year.

In-service Support

Synopsis:  The audit team reviewed the draft Service Level Agreement (SLA) dated February 7, 2011 which defines the service partnership between Natural Resources Canada and Agriculture and Agri-Food Canada (AAFC) for the provision, by AAFC, of Integrated and Financial and Material System services, Project Systems and system support, using SAP commercial software. The review focused on the post implementation period for the operation of the new SAP solution. Audit briefings to present the key findings and analysis of the review were provided to key project personnel and to senior management.

The purpose of the review was to ensure that all aspects of the draft SLA needing additional detail were clearly identified before finalization of the agreement between the two partners. The audit team reviewed each of the sections outlined in the SLA and provided comments on those sections requiring additional information or clarification. The review included the in-service support model to ensure that it was comprehensive, included governance, service and operating level agreements with the service provider, coordinated support functions and services, was fully pre-tested and adjusted prior to Go-live. The review also ensured that the SLA identified the need for fully trained and qualified staff.  

Audit Branch briefed management regarding items requiring additional information or where some clarification was needed and adjustments to the SLA were made accordingly to address these items.

Conclusion

By agreement between NRCan and AAFC -  the SLA established does not constitute a legal contract, but does contain the essential information necessary to establish an administrative agreement between two Federal government organizations. The SLA reviewed during the audit contained 33 sections. The administrative scope of coverage was extensive, and for the purposes of audit review each section was reviewed as an independent clause. However, due to the wide range of topics addressed and the specific nature of each topic, the audit did not establish a general conclusion for the SLA.

Operational Agreement at ‘Go-live’

Synopsis:  The audit team reviewed the draft Operational Level Agreement (OLA) for Incident, Service Request and Problem Management (between NRCan and AAFC) for evidence that the post-Go-live in-service support model for SAP was comprehensive, included key requirements, speed of service was defined (where required), and the responsibilities of both partners were clearly defined.

The purpose of the review was to ensure that all aspects of the draft OLA needing additional detail were clearly identified before finalization of the agreement between the two partners. The audit team noted that the OLA outlined the ‘processes, responsibilities and interactions between AAFC and NRCan associated with resolving incidents, service requests and problems related to the SAP Solution.

Audit Branch briefed management regarding items requiring additional information or where some clarification was needed and adjustments to the OLA were made accordingly to address these items.

It is acknowledged that the processes presented in the OLA will be modified and adjusted over time.

Conclusion

The OLA reviewed during the audit contained 30 sections. The scope of coverage was extensive, and for the purposes of audit review each section was reviewed as an independent clause. At the time of the review, there were no significant areas outstanding or incomplete. However, due to the wide range of topics covered and the specific nature of each topic, the audit did not establish a general conclusion for the OLA.

Looking to the Future

One of the general ‘lessons learned’ from SAP implementations is the importance of maintaining the Enterprise Resource Planning (ERP) methodology after the application has been implemented.  The original ERP methodology involves a thorough system analysis and business design to ensure that any changes to the application are aligned with the ensemble of active system processes.  Generally, after approximately 6 months in the post-implementation period, when users have had time to become familiar with the application and begin to embrace its advantages, there is usually an emergence of enhancement requests.  Often, instead of being implemented as an integrated component of the original ERP methodology using the original project disciplines, these changes are treated as independent fixes or enhancements.  By making individual system changes, a risk is introduced to the system of losing its original integrityFootnote 2 as implemented.

Change Management

Synopsis:     Critical SAP application and NRCan business change management objectives, respectively, were met for the April 4, 2011 ‘Go-live’ of SAP consistent with the Felix Blueprint.    Due to the limited scope of the Blueprint, however, significant change management objectives and challenges remain for the post ‘Go-live’ period in the areas of governance, application development, information-sharing and records management.

Ongoing SAP Application Changes

While the project completed the essence of the SAP development goals and objectives identified in the Felix Blueprint for the April 4, 2011 ‘Go-live’, there remains a significant number of minor fixes and user requested developments to be formally retained and carried forward for consideration into the post ‘Go-live’ period.  Included in these sources of potential system changes are: the Felix Opportunity Log, listing 20 possible improvements to SAP functionality that were set aside as not specifically identified in the Felix Blueprint; outstanding SAP Testing Defects, listing a small number of minor Quality Assurance (QA) defects,  User Validation Testing (UVT), a Blueprint requirement not performed that would likely have identified a number of minor to moderate application issues that will now have to be addressed in the post ‘Go-live’ period; and User Acceptance Testing (UAT), not comprehensively conducted for the Project System (PS) Module, likely resulting in a number of minor to moderate application issues that will need to be addressed in the post ‘Go-live’ period.

Ongoing Business Change Management

The partnership developed between NRCan and AAFC for operation of the shared enterprise-level Integrated Financial and Materiel System (IFMS) has resulted in a department-level Service Level Agreement (SLA).  The SLA is supported by four Operational Level Agreements (OLAs), including Incident and Problem Management, Release Management, Change Management and Training. The implementation of SLAs and OLAs has, in turn, identified the need for NRCan to clarify its internal governance and management processes to adapt to the new partnership arrangements. 

The NRCan IT Help Desk developed new operating processes and a new tracking function for SAP problem tickets for the April 4, 2011 ‘Go-live’ of SAP.  These operating processes, in part integrated with the AAFC IT Service Desk, were the result of readiness planning due to the completion of an Operating Level Agreement (OLA) between NRCan and AAFC.  The Service Desk practices and procedures can be expected to take some time to mature, especially given complexities of inter-departmental operations and increasing user demands such as problem diagnosis, data extraction, and reporting over the first year of SAP operation.

At the time of the audit, the graphical user interface (GUI) for SAP – the software that runs the on-screen image of SAP – was not fully compatible with Windows 7, the operating system for NRCan desktop computers.  NRCan is currently in a 3-year roll out of Windows 7 (Windows 7 is replacing Windows XP which is no longer supported by Microsoft).  Consequently, during this 3-year rollout, some SAP users trained in SAP that runs on Windows 7 may find that computer screens do not match the GUI on their home office PC (XP).  Additional guidance will be required.

Key SAP reference material for trainees was originally posted on a Felix Training SharePoint site. This site was used to post key operational and training materials for user reference.  The Audit Branch has been advised that subsequent to the audit, the materials on the site are to be archived, the site decommissioned and that future training documentation would be provided by AAFC. 

In respect of information and records management requirements, information and documents created by the Development Team and by the Business Change Management Team will continue to have an ongoing value as project information for the operational groups of the two departments. Arrangements will be needed with NRCan Records Management to maintain all of this information as a separate and complete NRCan information holding for future reference, whether for operational support, departmental management, central agency review, external audit, ATIP or other purposes.

Conclusion

Implementing SAP has resulted in a broad range of change management activities impacting, among other areas, user support, technology management and support, information management, and records management.  The business change management effort was challenged by the need to restructure technology support functions, by the identification of some Windows 7 and XP inconsistencies, by new requirements for information sharing, and by ongoing requirements for records management.  Further, a number of moderate and low priority requirements, and other user requests from the period prior to ‘Go-live’ - need to be formally retained and catalogued along with new year requests and requirements.

The sum of these challenges will have a significant impact on the transition of business operations from legacy systems to SAP. Consequently, each change management activity in support of the departmental transition to an integrated financial and materiel system will need to maintain a high degree of due diligence to effectively support business changes.

As identified above, a number of SAP implementation challenges remain. Given the extensive amount of ongoing development with the AAFC-NRCan amalgamated  SAP Centre of Excellence, on-going training, hardware and software adjustments, business transformation, application and business stability efforts and related communications required this year, it appears the $1,630,000 budgeted for Felix Project implementation (2011-2012) may only support a portion of the further costs to the department. The original budget is provided below. If ongoing SAP-related development costs prove to be significant, this could represent an unanticipated level of effort that the Department will have to address in the foreseeable future. The mechanism by which these new costs would be addressed will be through the costing arrangements outlined in the Service Level Agreement and various Operational Level Agreements.

The Project CharterFootnote 3 for Felix identified the 4-year cost estimates for the initiative by cost categories as follows:

The Project Charter for Felix identified the 4-year cost estimates for the initiative by cost categories as follows:
Project Cost Category Estimated Cost FY 2009-10 Estimated Cost FY 2010-11 Estimated Cost FY 2011-12 Estimated Cost FY 2012-13 Estimated Total Cost (000’s)
Project Office 1305 972 60 0 2337
Design & Planning 528 1041 0 0 1569
Development (Configuration) 124 3450 0 0 3574
Training 0 924 572 0 1496
Communications-Transformation 158 595 40 0 793
Support Planning 0 228 544 191 963
Project O&M:
Office Facilities 563 845 197 0 1605
Desktops, Tools & Support 8 159 10 0 177
Servers, Installation & Support 7 119 0 0 126
SAP License & Cluster Fees 0 300 0 0 300
TOTAL DIRECT 2691 8634 1423 191 12939
Project Contingency 393 1221 207 28 1849
TOTAL PROJECT 3084 9855 1630 219 14788
Corporate and Operational Reporting using SAP

Synopsis:     76 SAP reports were identified as available to SAP users in NRCan, beginning April 4, 2011, to meet key financial, materiel and project reporting requirements across the Department.  Compared to the previous legacy system, departmental business operations will find this a significant and challenging reduction in reporting capability.  In this regard, a departmental business intelligence strategy and plan are to be developed in the new fiscal year.

The Felix Blueprint identified 99 reports for development consideration for the April 4, 2011 ‘Go-live’.  The 99 reports were broken-down as 41 standard SAP reports, 14 reports to be developed using the SAP Report Painter tool, 8 custom-developed reports for the Project System (PS) module, 4 Cross-Application Timesheet (CATS) reports, and 9 Financial Management Report (FMR) requirements.   As of the April 4, 2011 ‘Go-live’, 76 reports were identified by the Felix Development Team as available and 23 Financial Management Reporting requirements were rejected as out-of-project-scope.

The departmental corporate reporting strategy and plan, as expressed in the Felix Blueprint, is that NRCan will rely on existing AAFC reporting functionality in SAP to commence in  2011-2012, in addition to customized reports that were prepared by the project for April 4, 2011.  However, NRCan will commence in 2011-2012 with no corporate enterprise database from which to populate and provide business information to the Department, as is in use at AAFC and has been in use at NRCan for its legacy systems.  Consequently, a departmental business intelligence (BI) strategy and plan are to be developed in the new fiscal year.

Conclusion

On April 4, 2011, given the reduction of financial, materiel and project reports and information compared to last fiscal year (FY 2010-11) legacy systems reporting capability, the Department can anticipate some user dissatisfaction with SAP reporting capability in the first year of operation and, consequently, may use other ad-hoc means to track expenses and free balances.

Appendix A
Lines of Enquiry

The following lines of enquiry were used in the audit:

  1. Change Management

    1.1     Change Management Support:  The extent to which departmental management provides the direction, decision-making, resource allocation and control needed to support the planning, development, testing and implementation goals of SAP in a timely manner.

    1.2     Business Process Changes:  Whether business process changes are fully identified and documented so that the Change Management Team can develop comprehensive training materials on both system functionality and the revised business processes the new system supports.

    1.3     Training Materials:  The extent to which training materials, both electronic and hardcopy (i.e., reference materials), are complete, comprehensive, illustrate both new system functionality and new business processes, and are drafted, revised, approved and released in sync with the Felix schedule and are available in both official languages.

    1.4     Train-the-Trainers:  Whether end-user and technical support trainers have the business and/or technical knowledge required to provide effective training.

    1.5     End-User Training:  The extent to which regions, sectors, branches, offices and groups send their primary resources for training, for the full training period, training occurs per the training schedule, and trainees are evaluated on the completeness of what they have learned. This includes the extent to which training meets user requirements.

  2. Reporting

    2.1     Identification of Corporate Reporting Needs:   Whether the corporate reporting needs of the Department have been comprehensively identified, vetted, documented and approved to the extent required, in order for the development teams to prepare for ‘Go-live’. 

    2.2     Data Availability:  Whether completeness, integrity and availability of SAP data will effectively support the Department’s overall data and reporting needs.

    2.3     End-User Support for Reporting:  The extent of formal end-user support for reporting is researched (e.g., benchmarked with OGDs), assessed, formally approved and resourced in order to meet departmental-wide reporting needs.

  3. Security & Authorizations

    3.1     Statement of Sensitivity (SOS):  Whether an SOS for the implementation of SAP is formally and properly completed per GOC standards and takes into account a formally completed SOS by AAFC.

    3.2     Threat and Risk Assessment (TRA):   Whether a TRA for the implementation of SAP is formally and properly completed per GOC standards and takes into account a formally completed TRA by AAFC.

    3.3     Preliminary Privacy Impact Assessment (P-PIA):  Whether a Preliminary PIA for the implementation of SAP is formally and properly completed per GOC standards.

    3.4     Network Certification & Authentication (C&A):   The extent to which a C&A for the implementation of SAP is formally and appropriately completed per GOC and NRCan standards and takes into account a formally completed SOS and TRA.

    3.5     Segregation of Duties:   Whether SAP Job Roles and Authorizations are formally completed and signed-off, and apply proper segregation of duties.

    3.6     Network and Infrastructure Security:   The extent to which departmental network and infrastructure risks, remediation, and monitoring and reporting have been identified, assessed, planned, resourced and tested in support of the implementation of SAP.

  4. In-Service Support

    4.1     In-Service Support Model:  The extent to which post-Go-live in-service support for SAP operates under a comprehensive support model including governance, MOUs and Service Level Agreements with AAFC, coordinates support functions/services, is fully pre-tested and adjusted prior to Go-live, and identifies the need for fully trained and qualified staff.

    4.2     Roles & Responsibilities:  The extent to which the post-Go-live in-service support model has roles and responsibilities fully documented and tested for the management and operations of primary, secondary and tertiary levels of support.

    4.3     Meeting User Demand:  Whether the projection for in-service support demand from NRCan, from all sources, is sufficiently researched (i.e., based on experiences of OGDs), assessed and reported as part of Felix planning.

    4.4     Hiring & Training Support Staff:  The extent to which the hiring and training of in-service support staff reflects the requirements of the in-service support model, roles and responsibilities, projected user demand for the service, and the Felix Plan and Schedule.

Appendix B
Audit Participants
Participant Sector Region
Director General CMSS/CRO NCR
Director (Business Design) CMSS/CRO/FP NCR
SAP Project Manager CMSS/CRO/FP NCR
Project Management Office Manager CMSS/CRO/FP NCR
Felix Planner CMSS/CRO/FP NCR
Design Team Manager CMSS/CRO/FP NCR
SAP Design Consultant CMSS/CRO/FP NCR
Technical Lead, Felix Project CMSS/CRO/FP NCR
Legacy System Coordinator CMSS/CRO/FP NCR
Director (Change Management) CMSS/CRO/FP NCR
Manager, Application Development CMSS/SSO/ITS/ITSUP NCR
Business Change Management Consultant CMSS/CRO/FP NCR
Manager CMSS/FMB/FS/FST NCR
A/Dir. Finance and Procurement Service CMSS/SSO/FAP/FIN NCR
Acting Director CMSS/FMB NCR
Departmental Assets Manager CMSS/SSO/FAP/PCAM NCR
Director, Financial Policy, Reporting and Internal Controls CMSS/FMB/FPRIC NCR
Manager CMSS/FMB/FPRIC/FMIC NCR
Manager CMSS/FMB/FPRIC/CR NCR
Policy Analyst CMSS/FMB/FPRIC/FMIC NCR
Manager, IT Security Program CMSS/IMB/EIAS NCR
Assistant Commissioner and Comptroller, Northern Pipeline Agency DMO/NPA NCR
Senior Policy and Program Manager DMO/NPA NCR
Administrative Assistant DMO/NPA NCR
Senior Director and Chief Technology Officer CMSS/SSO/ITS NCR
Manager IT Operations CMSS/SSO/ITS/ITIFA NCR
ABAP Team Lead Agriculture and Agri-Food Canada NCR
Project Systems Analyst CMSS/CRO/FP NCR
System Security Administrator, Financial Management Application (Saturn) Agriculture and Agri-Food Canada NCR
Appendix C
Appendix D