To provide information on how evaluations are planned, costed and conducted once they are on the five-year departmental Evaluation Plan.
The Cycle for Evaluation Reports
- Evaluation Assessment Evaluation Assessments are prepared based on:
- program profiles (e.g., objectives; logic models; organization and governance; expenditures; an assessment performance information);
- the calibration of the evaluation based on risk criteria, as part of overall 5-year Plan
- (i.e., program renewal; materiality; context and need; visibility; management practices and structure; policy and delivery complexity; performance measurement; and past evaluation and audit findings);
- the evaluation questions (TBS Policy identifies the generic questions and these may be supplemented by others identified by Program);
- an initial plan on how to conduct the evaluation (i.e., methods to be used and levels of effort; contracting strategy; timelines; and estimated costs).
- Evaluation Assessments are developed based on experience and professional judgement.
Based on evaluation assessment, the scope and costing are developed, taking into account:
- TBS policy’s minimum standards and required "multiple lines of evidence" (e.g., documents; interviews; administrative data; survey results; case studies; focus groups).
- SED’s internal risk assessment based on additional methods support quality, rigour & richness of the evaluation.
- Sufficient level of evidence to be collected to allow conclusions to be drawn on each program (especially G&C programs).
- There are many risk-factors (previous page) and considerations, e.g.:
- number of programs, availability of data, past evaluations;
- enough interviewees to ensure the full story;
- in-person interviews are always better than over the phone surveys provide less in-depth information from many people;
- case studies provide in-depth knowledge of one project and how projects contribute to the achievement of program objectives.
- Need for contracting
- Contractors may be used as part of the evaluation team, based on:
- The need for subject matter expertise;
- Internal capacity and timing considerations;
- and The need for third party (non-NRCan) involvement
- Value for money (optimal mix of internal and external resources) is determined through:
- The availability of internal staff to meet coverage requirements;
- and The competitive procurement process designed to achieve the best value for the Crown
- Evaluation has a Supply Arrangement with nine evaluation firms* based on a competitive process to qualify the firm and contractors.
- Based on the TOR, an RFP is sent to a minimum of 4 firms.
- Contracting takes at least 3 months from SOW to signed contract.
- Responses to the RFPs, provide suggestions for altering the SOW and what the company is prepared to do for the available budget.
- If a subject matter expert is required, Evaluation will contract directly with the expert to be part of the team (e.g., nuclear expertise).
* KPMG; Science Metrix; PMN; PRA; Goss Gilroy;TDV Global; Boulton; Baastel; and CPM.
Terms of Reference
- The TOR are derived from the Evaluation Assessment and scoping/costing analysis.
- TOR must be approved by the Evaluation Committee.
- TOR include:
- overview of the entity being evaluated;
- evaluation issues and questions,
- methods (e.g., interviews; surveys) to be used;
- contracting approach (i.e., in house; contracted out; or hybrid);
- resources required (e.g., estimated contract cost); and
- governance (e.g., a working group of program and evaluation officials).
- Evaluators work on several projects simultaneously.
- Field work presents many challenges:
- field work requires the input of programs;
- contact information for surveys & interviewees not always readily available;
- unplanned delays are very difficult for contractors to manage;
- work around seasonal cycles: e.g. interviewees not available during the summer; contractors and programs are extremely busy prior to March 31.
- Technical reports are usually produced for each method (e.g., interviews). All information is analyzed by evaluation question.
- Preliminary findings are presented to programs to confirm findings and seek any additional information.
Report and Recommendations
- Preliminary findings serve to highlight and validate key issues from collected evidence, and provide basis for the outline for the report.
- Based on the findings, the report is written and the recommendations are developed.
- The draft report is vetted with the Program and discussions take place on the recommendations (do they flow from findings; do they make sense; can they be implemented).
- Report length is influenced by complexity of subject (e.g., one program or many) and need to present evidence.
- Management responses and action plan are drafted by the program and ADM approval is sought.
Approval & Posting of Reports
- The DEC advises the Deputy Minister (DM) on the report and management responses action plans. The DM must approve.
- DM-approved reports are provided to TBS and may be examined for quality during the annual MAF assessment of evaluation.
- The Evaluation Policy requires, complete, approved evaluation reports along with management responses and action plans to be posted in both official languages in a timely manner (i.e., 90 days).
- Reports are reviewed by ATIP and Communications.
- If required, Communications and the program prepare media lines.
Management Responses & Follow Up
- ADMs are responsible for implementing the action plans for each recommendation.
- The Evaluation Division follows up with Sectors on the implementation of action plans and updates DEC.
- DEC decides if the management responses have been satisfactorily implemented and when the file can be closed.
- The activity/program is re-considered as part of Evaluation Plan in the following evaluation cycle.
What Does Evaluation Add?
- Accountability (posted reports and input into DPRs)
- Strategic input into program decision-making and development (e.g., assistance in developing logic models and performance measurement strategies) including information for MCs and TB Submissions
- Neutral perspectives and additional data collection methods
- Evidence for internal reviews (Strategic Reviews/SOR etc.)
- Corrective change, starting before the evaluation is over
- Follow up on recommendations
- We hope this presentation has provided a better understanding of how evaluations are planned, costed and conducted.
- Questions? send to firstname.lastname@example.org