Evaluation

Reports 2006

Minerals and Metals Sector (MMS) In-House Language Training Program (IHLTP) - Formative Evaluation (EV0604) October 23, 2006

1.0 LIST OF ACRONYMS

SSO Shared Services Office
PSC Public Service Commission
CANMET Canada Centre for Mineral and Energy Technology
CETC CANMET Energy Technology Centre
CB Communications Branch
MTB Mineral Technology Branch
IHLTP In-House Language Training Program
NRCan Natural Resources Canada
MMS Minerals and Metals Sector
MMS-OADM Minerals and Metals Sector; Office of the Assistant Deputy Minister
MMS-EIFAB Minerals and Metals Sector; Economics, Investment and Fiscal Analysis Branch
MMS-IABDB Minerals and Metals Sector; Industry Analysis and Business Development Branch
MMS-EXB Minerals and Metals Sector; Explosives Branch
MMS-PD Minerals and Metals Sector; Programs Division
MMS-PPB Minerals and Metals Sector; Policy and Planning Branch
MMS-MTB Minerals and Metals Sector; Mineral Technology Branch
MMS-MTB-ODG Minerals and Metals Sector; CANMET Mineral Technology Branch; Office of the Director General
MMS-MTB-MMSL Minerals and Metals Sector; CANMET Mineral Technology Branch; CANMET Mining and Mineral Sciences Laboratories
MMS-MTB-MTL Minerals and Metals Sector; CANMET Mineral Technology Branch; CANMET Materials Technology Laboratory
EPS Energy Policy Sector
EPS-EPD Energy Policy Sector; Environment Policy Division
ESS Earth Sciences Sector
ETPS-CETC Energy Technology and Programs Sector; CANMET Energy Technology Centre

Top of Page
1.1 BACKGROUND

Natural Resources Canada (NRCan) and its various Sectors work in compliance with the language requirements established by Canadian federal government legislation. For example, in order to help its employees, NRCan's Energy Sector published a reference guide dealing with language requirements in the workplace. This reference guide outlines the responsibilities and requirements set out in the Official Languages Act with which the employees must comply. The document also reminds employees that these requirements apply primarily in the National Capital Region and, more specifically, for certain organizational duties and positions. Some branches or research centres must ensure a certain level of bilingualism as they work with the public or with clients who communicate in both French and English. However, within the Minerals and Metals Sector (MMS), the CANMET Energy Technology Centre (CETC) is except from these requirements. In fact, the CETC offers services to limited, identifiable, external clients whose language of choice is English. As a result, the Centre is not obliged to comply with the language requirements.

In 2000-2001, NRCan performed a detailed audit of the Official Languages Program – Language of Work component. The audit revealed that monitoring and follow-up measures related to the Language of Work component needed to be developed. The audit report presented eleven recommendations for the use of official languages by employees of the Department as well as for third parties offering services and activities on the Department's behalf. 1 It was within this context that the Internal Language Training Program (IHLTP) was created in order to meet the development and language skill retention needs of MMS staff.


Top of Page
1.2 The In-House Language Training Program (IHLTP)

The aim of the IHLTP is to develop and improve upon the bilingualism of employees working for MMS by offering second language courses as well as various workshops and language training services. The Program was introduced as a pilot project in September 2002. The first courses were initially offered on a voluntary basis to employees of the Mineral Technology Branch (MTB) and, as of September 2004, to all MMS employees, in addition to MMS's Energy Policy Sector. In January 2005, Communications Branch joined the Program and CETC joined in April 2005. The Earth Sciences Sector also participated in the Program as a pilot project.

According to IHLTP employees, the Program has shed a positive light on language training and bilingualism in MMS. The objectives of the Program do not include upgrading or specific training for employees to help them pass exams held by the Public Service Commission (PSC). Rather, it offers a stimulating learning environment that encourages the constant use of the second language, in addition to improving the language skills of employees. Moreover, the IHLTP may have a positive impact on the organizational culture and may create a setting that reflects the objectives of bilingualism within the federal government. The Program was recognized by the Working Group on Official Languages as an exemplary practice within the Department.

IHLTP's unique features include its interal delivery, universal voluntary access and flexibility (e.g., temporarily withdrawing from the training).  Regardless of classification, linguistic profile or language requirements of the position, any employee of the participating Branches who has obtained the approval of his or her supervisor may register. In return, the employees agree to work an additionnal and a half hour for every three hours of training received per week. Furthermore, the adaptability of courses to the needs and demands of the students, the moderate intensity and low stress level that come with learning directly alongside peers, and the integration of the cultural aspect of the language in the training add to the distinctive elements of this Program.


Top of Page
1.3 Program Description

The Program's main training activity consists of French and English second language learning courses. In terms of participation, 75% of students registered for the French course since the Program was created. Groups of five to eight students are formed, taking into consideration the skill level of each student (up to 13 different levels are offered for English classes and up to 14 for French classes). When new students register, their profiles are established through the use of an oral placement test conducted by a teacher. The questionnaire used is based on the Program and the objectives established for each of the levels. Following this assessment, the teachers meet and divide the students into levels. After two weeks, the teachers meet again to readjust the groups (if necessary).

Sessions are offered in the fall, winter, spring and summer. However, English language courses are not offered in the summer. Each session lasts ten weeks and the majority of the courses are offered in a three hour-per-week format that takes place during regular business hours. The course schedule and objectives can be adapted to meet demand and learning needs of the students.

In addition to its main activity (second language training), the Program also offers four types of services that include:

  • coaching for employees who must give a speech or a presentation in their second language;
  • workshops on specific topics;
  • private courses; and
  • cultural activities.

Information concerning these complementary services was not included in the data collection and analysis for this evaluation because of their relatively recent addition to the program mix and the informal delivery of many of these services.


Top of Page
1.4 Logic Model

The Logic Model presented in Figure 1 was developed for this evaluation. The model defines the Program's resources and main activities, outputs and results.

Figure 1: Logic Model for the In-House language Training Program of the Minerals and Metals Sector at NRCan - Chart Format

(Text Format)

Figure 1: Logic Model for the In-House language Training Program of the Minerals and Metals Sector at NRCan

Figure 1: Logic Model for the In-House language Training Program of the Minerals and Metals Sector at NRCan - Text Format

(Chart Format)
Organizational Development Advisor


Branch Supervisors (clients)

Workplace Resources


Program Manager - Full Time (1/3 French teacher)


French teachers (3) - Full time

English teachers (2) - Part time

Preliminary activities and activities between sessions

  • Evaluate acquired skills of students
  • Prepare classes
  • Evaluate new registrants
  • Establish student groups
  • Integrate cultural aspect
  • Adapt objectives according to demand and student needs
  • Manage operations
  • Develop training and monitor quality
  • Ensure students are offered flexible participation
  • Ensure that services offered are low-intensity and low-stress

Activities/Outputs


Second
language courses English/French


Training for presentations and speeches


Workshops on specific topics


Private courses


Cultural activities


Internal evaluation of training


Development, communication and program's reach

Short-term Results (1 to 3 years)


Improve verbal and writing skills and understanding of the second language (Student(s) - Employee(s))


Improve the level of confidence in the second language (Student(s) - Employee(s))


Maintain the competency acquired and/or pass Public Service Commission (PSC) (Student(s) - Employee(s))


Increase language skills of staff in the Sector (Branch/Sector)


Emphasize the importance and promote a positive image to language training biligualism (Branch/Sector)

Medium-term Results (4 to 7 years)

  • (Student(s) - Employee(s))
  • Increase productivity
  • Increase office morale
  • Increase tolerance for the use of both official languages
  • Increase employee commitment to the organization
  • Create a positive impact on employee opinion regarding management's commitment to employee development

Professional and personal objectives (Student(s) - Employee(s))


Encourage and increase the daily and continued use of the second language (Branch/Sector)


Expansion of the IHTLP to other branches within the Department (Branch/Sector)


Change in organizational culture through language training (Branch/Sector)

Long-term Results (8 years +)


Biligualism within the federal government


Top of Page

2.0 IN-HOUSE LANGUAGE TRAINING PROGRAM RESOURCES

According to the data provided by program management, the program received $315,480 from participating branches in 2005-2006. Over the course of this year, the Program's operating budget was $310,500, which is allocated as follows: $240,000 for salaries, $48,000 for employee benefits and $22,500 for educational material and program delivery. IHLTP annual budget is about $350,000. Funding is provided by participating Branches in accordance with a cost recovery policy. The formula is based on the participation rate over the total number of employees from each Branch rather than fees per participant. Therefore, the costs are considered as an investment into a program that benefits the entire organization. This formula ensures program access and makes managing the Program easier. To ensure that employees do not sacrifice the opportunity for professional language training, the Program does not receive funding from the Employee Training Fund.

Those involved in providing training are employees of the Department. The courses are taught by five teachers, all of whom are supervised by a full-time project manager who divides her time between management (2/3) and teaching (1/3). The manager/teacher, two full-time French as a second language teachers and one full-time English as a second language teacher all work at one site (Booth Street) in Ottawa. One full-time French as a second language teacher is assigned to the Bell's Corners site in Ottawa and a part-time English as a second language teacher is based in Val d'Or, Quebec. The Program also offers video teleconference courses for certain students living outside these regions. The geographic distribution of the teachers allows for the delivery of courses that are better adapted to the students and provides the teaching staff with a feeling of belonging. In turn, the teaching staff is more committed and gets involved in the Department beyond just language training.


Top of Page

3.0 EVALUATION OBJECTIVES AND SCOPE OF WORK

The formative evaluation of the Minerals and Metals Sector's In-House Language Training Program (IHLTP) was performed between February and May 2006 and the period covered is from fall 2002 until May 2006. The evaluation focused on relevance, design, delivery, cost-effectiveness and level of success.


Top of Page

4.0 STUDY CONSTRAINTS AND LIMITATIONS

  1. The IHLTP began as a pilot project in September 2002 and its current structure is relatively new. In addition, several of its activities are delivered informally;
  2. This evaluation represents the first analysis of the In-Housel Language Training Program and as such, accountability and audit frameworks have yet to be developed. The capacity to conduct comparative analyses (pre/post) was therefore limited;
  3. An electronic and standardized database containing data on participation, attendance and finances as well as internal evaluation results on acquired skills did not exist. Therefore, very little information was available and collected on the activities, deliverables and results (impact) of the Program;
  4. This evaluation did not measure the quality of the training activities nor student skill and language acquisition levels;
  5. The analysis on cost-effectiveness was limited to brief cost comparisons with other language training programs as well as existing studies and data available.

Top of Page

5.0 METHODOLGY

Three data sources were used as part of the analysis of this Program; two from primary data sources (Internet survey of 309 students, 181 of whom were reached for a participation rate of close to 60% and 11 telephone interviews with key respondents) and one secondary data source (documentation review).

5.1 Documentation Review

Various documents were reviewed, including operational, administrative, decision-making and financial documents. The following are the key documents and data reviewed:

  • the content of the Program's internal Internet site, electronic communication and official documentation used to present and promote the Program;
  • the description, lesson plans and training material for the second language;
  • internal evaluations completed by students;
  • the documentation relating to the daily management of the Program such as job descriptions for program employees, statistical reports concerning student participation and attendance, as well as documents that provide an understanding of Program activities;
  • a proposal to expand the Program department-wide; and
  • detailed lists of contacts, including the students, employees and Program Manager, as well as student supervisors listed by Division and Branch.

Top of Page
5.2 Student Internet Survey

A survey for NRCan employees having participated as students in the Program was conducted via the Internet. Each of these employees received an e-mail in which they were given a link to access the survey's questionnaire that was available in both official languages.

The objective of the questionnaire was to gather quantitative data on the majority of questions that are the focus of this analysis. The number of students surveyed (n=309) included all those having participated in the IHLTP training and covered all 11 Branches that are participating, or have participated, in the Program. The Mineral Technology Branch (MTB) represents the largest participant source (38%).

In total, 309 individuals, 230 of whom took the French language training and 79 of whom took the English language training, were invited to complete the survey. This sample consists of students who are either currently registered in a course or former students. With the exception of the employees who left MMS or the federal public service, this group represents practically the entire population of students who could have been contacted through the survey.

The internet survey was carried out between March 22 and April 13, 2006. The response rate was 59%, or 181 responses out of 309 invitations to participate. Of the total number of respondents, 40 are taking or have taken the English course and 141 are taking or have taken the French course. This distribution is practically the same as the entire population of Program participants. A total of eleven Branches are represented in the survey, 7 of which are within MMS and 4 from other sectors. Finally, close to 62% of respondents were in the process of taking a training course during the survey period and close to 30% were former students.


Top of Page
5.3 Telephone Interviews with Key Informants

Data was also collected from staff and managers of participating Branches. The telephone interview guide was administered in either official languages, according to the preference of the respondents. In total, eleven key informants were interviewed, including:

  1. The full-time Program Manager (1) who divides her time between management and teaching;
  2. Two second language teachers, one for the English language, the other for the French language;
  3. Nine (9) managers from Branches or Divisions who gave their employees approval to participate in IHLTP courses and who followed their progress. In addition, some of these managers are currently taking, or have taken, courses offered by the Program.

Top of Page
5.4 Integrated Analysis

Following the data collection, the quantitative and qualitative data were analyzed, organized and integrated according to subject and evaluation questions. Data coming from different sources was analyzed using a triangulation approach. Triangulation involves using various sources of complementary data in order to verify the convergence of the results for certain questions or to identify incomplete or contradictory results. An integrated results analysis therefore allowed strong support for reviewing the Program results and for formulating recommendations.


Top of Page

6.0 RESULTS AND CONCLUSIONS

6.1 Relevance

Several elements which establish the Program's relevance were identified. These include a particularly high proportion of students and managers expressing their satisfaction with the language training Program, the presence of a significant and constant demand and the attendance of participants. The survey also revealed that the respondents feel that the Program does a good job of meeting the objectives and needs of managers, employees and, overall, the various Branches (e.g., its ability to adapt to the specific needs of its clients, etc.). The fact that few alternatives to the IHLTP exist and that the objectives and needs met by the Program are different from those addressed by external programs only serves to strengthen the Program's merit and usefulness.

The features, objectives and needs associated with other types of available training are significantly different from those of the language training Program. As reported by several stakeholders questioned during the investigation, in contrast to external training programs, this Program is designed to introduce all employees to the basics of the second language, to promote a positive impact on the performance and use of both official languages in certain Branches and, finally, to facilitate the retention of language skills. However, the Program is not designed to quickly produce bilingual employees who can pass exams held by the PSC. That is not at all the objective of the Program. In other words, the Program cannot replace the intensive training offered externally. The Language Training Program acts as a tool that is complementary to external training programs and provides an opportunity to satisfy the needs that are not covered by that type of training. Uniting these different approaches maximizes the integration and use of both official languages within the various Branches. The complementary nature of the Language Training Program, its unique aspect and the absence of alternatives represent substantive arguments for justifying the Program's existence and affirming its relevance.

The evaluation revealed that the Program did an excellent job of meeting the needs of beginner or intermediate level employees and, although it cannot fully meet the needs of advanced students, it allowes students to retain their language skills. The Program therefore meets a specific need that is not covered by intensive training offered externally.


Top of Page
6.2 Design and Delivery

In-house delivery, accessibility (Universal Access), and the Program's flexibility have been identified as the key elements to the IHLTP's structural success.  As a matter of fact, respondents emphasized that the Program integrates well with their work given that it facilitates access to second language courses regardless of the language skills or requirements of the employee's position. It also encourages continual learning and optimizes the use of time dedicated to work and training.

Among the positive effects associated with the fact that the Program is offered internally, students emphasize that it promotes accessibility to second language courses, maximizes their course time in comparison with the travelling time required for external programs and facilitates the organization of work. Students feel that the fact that the teachers are also Department employees and that they teach in their own work setting had a positive impact on their attendance, their motivation to learn a second language and their actual learning through close contact with peers and teachers.

Like the students, managers consider the internal nature of the Program to be one of its positive elements. The proximity of the teaching staff with the employees going through the training, as well as the cost-benefit of the Program have been identified by managers as two positive effects of the In-House language training.

The overall level of satisfaction with the teachers and the design and delivery of the courses offered is very positive. Specifically, the existence of clear objectives, the adequate distribution of students and group sizes, as well as a follow-up process for students and managers facilitate participation, learning and progress while limiting the stress level. Teaching material and methods are generally satisfactory.

However, one aspect that seems to garner unanimous agreement from students, managers and teachers is that courses held via video teleconferencing are not successful. In addition to a low rate of participation, which suggests that this method is considered inadequate, several students contend that this learning method is not very easy or effective. In fact, the presence of an on-site teacher increases interaction and offers opportunities to ask questions or to practice the second language with the teacher outside of class. Such benefits do not exist with video teleconferencing.

With regard to course content, it seems to be well-adapted to the objectives, demands, language needs and schedules of the students. However, some respondents mentioned that the Program is effective for beginner and intermediate learners, but that it only facilitates skill retention for advanced learners.

Despite that fact, respondents generally consider the Program to be efficient and well-managed. On the other hand, several of them feel that the IHLTP would benefit from being better known among employees and management, particularly in certain Branches. A number of respondents also believe that the Program would benefit from additional resources as a means of improving its offerings. Furthermore, the majority of respondents are in favour of the Program's expansion within the Department. However, this would have to be done carefully in order to ensure that the Program's unique advantages remain unchanged.


Top of Page
6.2.1 Reimbursement of Part of Training Time with Work
Hours

The Program requires that participants make up half of the time spent in class during normal business hours. This initiative stems from a decision made by senior management that seeks to distribute the investment among managers and employees since the funding reserved for the IHLTP is separate from funding normally used for language training. This acts as a means of ensuring the motivation of the employees registered in the Program as well as ensuring that their work-related performance levels will be maintained despite their absence during normal business hours.

For every three hours (180 minutes) of course time, students are supposed to make up an hour and a half (90 minutes) in overtime. This condition of participation appears in the form of an agreement when students register for courses. However, although it is integrated into the Program, compliance with this condition is at the supervisor's discretion and is not monitored by the Program. IHLTP management and teachers do not handle this matter.

Still, it should be noted that although the survey respondents are, for the most part, in favour of the employee participation model requiring the make-up of a part of the time spent in training, this model can seem unfair as it is often applied differently by different managers.

For their part, a majority of the managers questioned are in favour of the employee participation model involving the make-up of training time. Two-thirds of them feel that the model is quite adequate. The following are their main reasons for this:

  • This model represents a mutual commitment of both parties and encourages employees to attend and get involved in the training;

  • The impact on work is not significant since the training only takes up three hours each week. It is therefore not difficult for the students to make up the 90 minutes;

  • It is normal that employees reimburse their work hours since they are taking advantage of these measures for professional and personal levels. In many cases, employees would have to take such courses in the evening.

The other third of managers questioned consider the time reimbursement model to be somewhat adequate or not very adequate. The following are their main reasons for this:

  • Employees already work many overtime hours and do not claim them. It is not appropriate to ask them to reimburse their time;

  • This model sends a contradictory message to those in management who support training for their employees since, on one hand, managers want to encourage their employees to develop their language skills and, on the other hand, those same employees are asked to make up their time spent in training;

  • Managers regularly send employees for training related to their positions or to career development and they are never asked to reimburse their time. This training is just like any other;

  • The managers do not want to be responsible for monitoring employees so that they make up their hours. Furthermore, this would have a negative impact on employee morale as well as on the participation and program drop-out rates.

It has been noticed that there is no consistency for reimbursing hours, even between employees who are from the same Branch. Moreover, there are employees who feel that this model is unfair since some managers require that those hours be reimbursed while others do not.

Of the nine managers questioned, only two have a formal agreement with their employees who register in the Program. Four of the managers who agreed that the reimbursement model was adequate have informal agreements with their employees participating in the training. Furthermore, they say that they are very flexible since the majority of their employees regularly accumulate unpaid overtime hours at their normal job. Considering the Program's importance for the Department, these managers feel that it could seem abusive to ask those employees to make up for time spent in class. For this reason, some managers have totally abandoned the request for obligatory time reimbursement from their employees participating in the IHLTP. These managers have chosen a more flexible model, making time reimbursement optional.

One year after the Program began, the IHLTP management raised the issue of time reimbursement and asked senior management whether this policy should still be in effect or abandoned completely. The response from senior management was to keep the time reimbursement policy in effect. However, in terms of this policy, senior management, the IHLTP Manager and Branch managers do not conduct follow-ups.

Student Opinions

From the student point of view, 54% of survey respondents feel that the time reimbursement model is quite adequate, while 23% feel it is somewhat adequate. Reasons given for being in favour of this model are mainly that it is easy to make up the hours considering the travel time saved (15 respondents) and that it is a win-win situation for both employers and employees (10 respondents). Nevertheless, 22% of students felt that this model was inadequate or very inadequate. In response to an open question, these students answered that they should not have to make up time (19 respondents) and that the time reimbursement model is a negative reflection of the value that the Department gives to second languages (9 respondents), particularly given that the government would like to support a bilingual work environment. Others stated that time reimbursement would increase an already heavy workload, thereby increasing the stress level of employees. Finally, some respondents mentioned that making up hours could conflict with the responsibilities that employees have to their families.

The range of opinions that students in each sector have on the time reimbursement model reveals that there is consistency in opinion between students from the MMS (98 respondents) and other sectors (78 respondents). In both cases, more than 75% of respondents feel that that this model is quite adequate or somewhat adequate while 20% feel that is inadequate or very inadequate. However, responses in favour of the time reimbursement model contradict statements made by managers and some students. In fact, this policy is not enforced across all the branches, and less than half of the students have made up their required number of hours.

In order to evaluate the relevance and scope of this reimbursement model, the students were asked how they make up the work that had to be done while they were in training. Of the 181 respondents, 65% (or 117 students) stated that they made up for the time spent in training by working overtime. Furthermore, 33% of respondents stated that they increased their productivity levels during regular business hours and 18% stated that they did not change how they use their time.

Given that only a few Branches apply this reimbursement model, not all employees participating in the IHLTP follow it. Managers do not seem to approach this issue in similar manners, even if they say that they have agreements with their employees. This significant variation was identified through the use of a small sample of managers (9). It is quite probable that this variation exists in the other divisions and branches that were not interviewed.

The policy concerning making-up a portion of the time spent in training does not seem to be applied in several branches due to a number of factors. Only 43% of survey respondents make up at least 90 minutes in overtime. However, 77% of survey respondents and 78% of managers indicated that this policy is quite adequate or somewhat adequate. In summary, people seem to approve of the idea but do not feel that they have to abide by it.


Top of Page
6.2.2 Number of Students per Group

In general, groups of five to eight students per class is appropriate. According to 68% of the students, the distribution according to the number of students was very adequate while 92% feel that, overall, it was satisfactory. 2 In response to an open question, several students emphasized the fact that offering the courses in small groups is beneficial because of the frequent contact with and level of trust among peers fosters a higher level of learning among the participants. Similarly, small group sizes ensure increased involvement, allow for more attention from teachers and create a less intimidating classroom environment. However, it is important to note that 11% of survey respondents quit the training because they found that group work did not allow them to meet their objectives.


Top of Page
6.2.3 Adaptability of Training

For students, another desirable feature is whether the course objectives are easily adaptable to their language needs. Essentially, over 94% of respondents feel that such flexibility is very important. When answering an open question, twenty-four students revealed that such a feature was a large advantage of the Program. When the students were asked about the implementation of this objective, they answered that the training adapted well. Eighty percent of survey respondents consider that the training meets their language needs. 3 When questioned about the cultural aspects of the language taught, 83% of respondents felt that cultural aspects need to be integrated into training and 80% felt that the Program successfully integrated those elements. However, it is important to note that only 39% of students feel that the implementation is 'very successful'.

The majority of students were satisfied with the way the course adapted to their positions. They found that the language training offered was particularly adapted to the context of their jobs, especially with regard to terminology and topics discussed during course time. However, teachers note that it is much easier to adapt advanced level courses to the workplace. At the beginner level, there is limited opportunity for this type of adaptation since the course focuses on fundamentals.

According to the teachers, the Program adapts very well to student demands, needs and problem areas. In general, teachers very rarely come across students who would like to modify the Program or course plan. However, at times, specific requests have been made, such as working with certain grammar points, learning a more technical vocabulary for job purposes, reading novels, or simply the opportunity to decide whether to watch a film, television or have a debate.

This ability to adapt has its limits since the teacher must consider the Program and the needs of the entire group. Moreover, situations where stronger students impose a course format that is more comfortable for them to the detriment of the other group members must be avoided. For these reasons, some respondents maintain that it would be preferable for the courses to have a more rigid structure This would standardize the course and students would find themselves at the same level.

Some survey respondents pointed out areas where they felt structural improvement is needed. Twelve students indicated that they found the flow of the Program to be too slow. Additionally, eleven of them stated that they would have liked more course time each week. Lastly, seven students indicated that they would have liked the groups to be selected more effectively based on the level of the students. Courses could then be structured in terms of levels. Regardless, these are minority opinions in comparison to those expressed by all the students.


Top of Page
6.2.4 Training Service Delivery

The overall level of satisfaction with regard to the design of the second language courses offered by the Program is very positive. In fact, 54% of respondents state that they are very satisfied with the courses they have taken and 44% are satisfied. 98% of students are satisfied with the Internal Language Training Program's content, method and learning activities. 4 Similarly, 85% of respondents indicate that the material used by the teachers to deliver the subject to the students are quality products.

When the students were asked to discuss their level of satisfaction with regard to the teaching offered in the second language courses, 72.9% of students responded that they were 'very satisfied' while 24.9% of participants were 'satisfied'. Overall, 97.8% of students are satisfied with the instruction offered by teachers at NRCan. 5

Among students taking the English as a second language course, 7.5% of them felt that the instruction was unsatisfactory. It is important to note, however, that in actual numbers, this statistic only represents 3 of 41 respondents having taken an English course. Furthermore, only 1 of the 141 French course students felt dissatisfied with the training received.

When the students were asked to name the main advantages of this Program, a number of participants indicated the quality of the teachers. Among other things, they emphasized the wide availability of the teachers, their enthusiasm as well as the interest in their work.


Top of Page
6.3 Program Administration, Development and Reach

Generally speaking, all managers feel that the Program is well managed and efficient as it improves the language skills of their employees and heightens their performance. Managers appreciate, among other things, written assessment reports that allow them to follow the progress of their employees and see the return on their investment; such feedback is not often available through external training.

In fact, one of the features of the IHLTP is to identify the learning objectives of each of the training levels offered. At the end of the session, teachers check if the objectives have been met by having the students take an oral exam. Having clearly defined objectives is very important to the majority of the students (close to 55% say very important, close to 40% say somewhat important). More than 88% of respondents experienced a high success rate, indicating a well-implemented program where set objectives are met. In fact, 48% of respondents feel that this aspect is 'very successful' while 40% feel that it is 'somewhat successful'. In response to an open question, fifteen students felt that this structure, which is based on reaching clearly defined objectives, is a strength of the Program. Among other things, they mentioned that the continuity, reliability and integration of the Program in the workplace strongly facilitate the attainment of program objectives.

When asked about the IHLTP's reach, the majority of the managers mentioned that the Program was visible, accessible and known by members of their organizations. Although some managers feel that the current communication strategy is very effective and functional and that it does not need to be improved in any specific manner, the majority of managers (6 out of 9) felt that the IHLTP's visibility among its current and future users, and management could be improved. Although e-mails and presentations are opportunities to inform and increase awareness of the Program, some managers still believe that a gap exists in terms of communications. In particular, some feel that the Program's visibility in certain Sectors and various locations within the Department needs to be increased. This evaluation did not measure the reach this Program has among NRCan employees who have not taken courses offered by the IHLTP. However, an analysis of the results from the Internet student survey reinforces this reasoning, as the IHLTP's reach seems to vary by Sector. While the Program is only 'well known' for 34% of respondents from other Sectors, 64% of respondents from the MMS indicated that the Program was well known in their work setting.

Data analysis shows that the Program offers services that are complementary to second language courses. In particular, the Program offers preparation workshops for employees who must give speeches in the second language and even offers 120 hours of private lessons to employees wishing to be adequately prepared to take the PSC exams. However, although these services and workshops seemed particularly in demand as they make it possible to meet the needs that had not been met with regular training, many managers questioned during the evaluation seemed to be unaware of these services. As indicated by one of the teachers, this is definitely due to the fact that the Program does not advertise these services simply because it does not have the capacity to meet the demand.


Top of Page
6.4 Cost-Effectiveness

Cost-effectiveness is also an important element of the Program noted by managers. According to managers, basing the financial contribution of each Branch on the total number of employees makes this Program cost effective. The IHLTP reduces the financial risks associated with student withdrawals, ensures greater flexibility and accessibility for employees, while not affecting the resource envelopes normally reserved for language training since the IHLTP receives funding from a separate source. However, it is still difficult to assess the cost-effectiveness and relevance of the IHLTP due the Program's lack of financial, follow-up or performance indicators.

This section is not an analysis of the cost-effectiveness based on a complete audit of the costs associated with Program operations. The key objectives of this cost-effectiveness analysis include:

  1. situating the cost of the Program's second language courses among those offered by external language training suppliers (private, public or academic institutions) and
  2. estimating the cost-effectiveness of universal access to the in-house Program.

An analysis of the fees of 24 private schools (specialized businesses) and public schools (colleges and universities) reveals that second language courses offered to groups cost, on average, $37 per student hour of class time. Some schools offer better costs for groups of students who commit for a specific number of course hours. Such rates vary between $12 and $25 per student hour of class time. Of the 24 teaching institutions, 90% of them offer the opportunity of holding courses where their clients work.

With regard to the IHLTP, the actual average cost per student and per session is $594 while the actual average cost per student hour of class time (10 week session consisting of 3 hours of course time per week) is $20. This data was taken from an analysis of fees for contributing branches for the 2005-06 sessions as well as the spring 2006 session (Table 1) and from a calculation of costs per student hour of class time (Table 2). Financial data and data relating to participation for the preceding fiscal years were not complete or standardized at the time the study was carried out.

Table 1: Amounts Charged to Branches Participating in the IHLTP Per Year and Session
Sector / Branch Total of
Employ-
ees 2005-06		 Total # of
Stud-
ents
2005-06		 Partici-
pation		 Participants
2005-06		 Costs
2005-
06		 Sp-
06		 Sp-
05		 Su-05 A-05 W-06 Sp-06 Total
2005-06		 Total
2006-07		 Per
Session
2005-06		 Per
Session
P-2006
MMS-MTB-ODG 22     36% 8 7 11 8 8 $5,000 $8,380 $1,250 $2,095
MMS-MTB-MMSL 148 86 29% 23% 36 17 36 37 35 $50,000 $58,700 $12,500 $14,675
MMS-MTB-MTL 125     12% 14 2 15 17 16 $43,000 $49,170 $10,750 $12,293
MMS-EXB 65 10 15% 13% 8 5 6 10 9 $18,000 $25,540 $4,500 $6,385
MMS-PD 50 14 28% 27% 15 6 16 13 12 $16,000 $17,150 $4,000 $4,288
MMS-EIFAB 23 6 26% 21% 5 3 6 5 6 $7,360 $11,050 $1,840 $2,763
MMS-IABDB 23 10 43% 16% 5 5 7 7 4 $7,360 $9,530 $1,840 $2,383
MMS-PPB 35 5 14% 13% 3 2 3 2 3 $11,200 $8,770 $2,800 $2,193
MMS-OADM 8 3 38% 0% 1 1 1 0 0 $2,560 $3,430 $640 $858
Total - MMS 499 134 28% 18% 95 48 101 99 93 $160,480 $191,720 $4,458 $5,326
EPS-EPD 90 37 41% 22% 20 12 20 25 18 $30,000 $32,000 $7,500 $8,000
CB 55 17 31% 11% 8 4 8 7 6 $18,000 $20,000 $4,500 $5,000
ETPS-CETC 200 67 34% 22% 45 30 46 47 43 $65,000 $68,000 $16,250 $17,000
ESS * N/A N/A N/A N/A 110   10 11 13 $42,000 $12,000 $10,500 $3,000
Total - Others 345 121 35% 18% 183 46 84 90 80 $155,000 $132,000 $9,688 $8,250
Total - All 1,343 389 30% 18% 278 94 185 189 173 $315,480 $323,720 $7,073 $6,788
*ESS participated in a pilot session in the spring of 2005 at a cost of $375 per student per session.

As of the summer of 2005, only French courses are offered through IHLTP services.


Sp = Spring

Su = Summer

A = Autumn

W = Winter
List of Acronyms

Source: Data provided by Program Management

 

Table 2: Actual Costs for Second Language Courses Per Student Per Course Time
Sector /-

Branch		 Actual Costs ($)
per Student per Session		 Actual Costs ($)
per Student per Hour
Sp-05 Su-05 A-05 W-06 Sp-06 Sp-05 Su-05 A-05 W-06 Sp-06
MMS-MTB-ODG 156.25 178.57 113.64 156.25 261.88 5.21 5.95 3.79 5.21 8.73
MMS-MTB-MMSL 347.22 735.29 347.22 337.84 419.29 11.57 24.51 11.57 11.26 13.98
MMS-MTB-MTL 767.86 5,375.00 716.67 632.35 768.28 25.60 179.17 23.89 21.08 25.61
MMS-EXB 562.50 900.00 750.00 450.00 709.44 18.75 30.00 25.00 15.00 23.65
MMS-PD 266.67 666.67 250.00 307.69 357.29 8.89 22.22 8.33 10.26 11.91
MMS-EIFAB 368.00 613.33 306.67 368.00 460.42 12.27 20.44 10.22 12.27 15.35
MMS-IABDB 368.00 368.00 262.86 262.86 595.63 12.27 12.27 8.76 8.76 19.85
MMS-PPB 933.33 1,400.00 933.33 1,400.00 730.83 31.11 46.67 31.11 46.67 24.36
MMS-OADM 640.00 640.00 640.00     21.33 21.33 21.33    
Total - MMS 490.00 1,209.00 480.00 489.00 538.00 16.00 40.00 16.00 16.00 18.00
EPS-EPD 375.00 625.00 375.00 300.00 444.44 12.50 20.83 12.50 10.00 14.81
CB 562.50 1,125.00 562.50 642.86 833.33 18.75 37.50 18.75 21.43 27.78
ETPS-CETC 361.11 541.67 353.26 345.74 395.35 12.04 18.06 11.78 11.52 13.18
ESS *                    
Total - Others 348.52 763.89 585.00 561.00 476.00 12.00 25.00 20.00 19.00 16.00
Total - All 419.00 986.00 533.00 525.00 507.00 14.00 33.00 18.00 18.00 17.00
*ESS participated in a pilot session in the spring of 2005 at a cost of $375 per student per session.

As of the summer of 2005, only French courses are offered through IHLTP services.


Sp = Spring

Su = Summer

A = Autumn

W = Winter
List of Acronyms

Source: Data provided by Program Management

Compared to the rates offered by private and public schools, the cost per hour of class and per student for IHLTP-contributing branches seems very competitive. In fact, the IHLTP offers training for approximately $20 per hour while schools outside the Department charge $37 on average.

This cost is calculated using the total contribution of the Branches participating in the Program converted to cost per hour of training time according to the number of participating employees. This formula only takes into account the second language course services offered by the IHLTP. In reality, the Program conducts several other activities or tasks beyond the hours spent in group second language courses, which are not included in the costs presented above. Within a 10-week session, management of the IHLTP estimates that the entire teaching staff spends more than 250 hours carrying out tasks beyond delivering second language courses to groups. Specifically, teachers allocate 25 hours for extra training assistance, 25 hours for assistance in preparing work documents and presentations in the second language, 120 hours to private courses without additional fees and 50 hours for participating in various committees and activities in the workplace (executive committee, official languages committee, social committee). Furthermore, between group course sessions, teachers allocate 18 hours to specific workshops and 5 hours to organizing cultural activities.

It is not possible to estimate the actual cost of only the second language courses, excluding the other training and support activities. However, an analysis of the data available seems to demonstrate that the Program offers a range of second language training services at a very competitive cost. Therefore, the IHLTP's service delivery seems quite cost-effective for participating Sectors and Branches.


Top of Page
6.5 Success and Impact

According to respondents, the IHLTP seems to have a relatively greater and more direct impact on the students than on the work environment. Although the training offered has a positive effect on the ability and confidence of students in expressing themselves in their second language, the evaluation shows that the Program has little impact on the use of the second language during formal and informal communication in the workplace. Similarly, respondents feel that the Program has a positive but slightly weaker impact on performance, employees commitment to employer, organizational culture and appreciation of Canadian linguistic duality.


Top of Page
6.5.1 IHLTP Impact on Students

Training offered by the Program has impacts on students in various ways, either at work, outside the office or on their awareness of Canada's linguistic duality. Ninety percent of respondents maintain that the Program has had a positive impact on their ability to communicate for work purposes (Table 3). In particular, the Program has had a positive effect on the level of confidence of the employees in using their second language at work. In fact, 85% of student respondents recognized that the IHLTP increased their confidence in expressing themselves at work in their second language. Based on student opinion, it is fair to say that the IHLTP clearly makes a beneficial impact on employee bilingualism. 6

However, survey results reveal a slightly less positive impact on various other work-related aspects. Indeed, 75% of respondents maintain that the training offered had a positive impact on their use of the second language in the workplace. The Program's impact on performance, morale and interpersonal skills, as well as the ability to work as part of a team are, however, not as positive. In these cases, the percentage of respondents who indicated a positive impact is in between 55% and 65%. The Program's impact on the short, medium and long-term objectives is relatively similar as 72% of respondents feel that the training would have a positive impact on their short-term objectives while 77% feel that way about their medium and long-term objectives.

The training offered had a less substansive impact on employee commitment toward the employer, and is one of the lowest percentages for a positive impact. Overall, 65% of respondents indicate a positive impact, 26% of whom express a very positive opinion. However, student opinion on the employer's commitment to human resource development is slightly more positive with 75% of respondents indicating a positive impact, 35% of whom feel it is very positive and 40% somewhat positive. The impact on the awareness, appreciation and tolerance of linguistic duality in the workplace is also considered positive as 70% of respondents indicate that the Program has had a positive impact.

With respect to the impact of training on the use of the second language outside of work, 68% of respondents contend that the impact was positive. However, only 20% of them see this impact as very positive.

In general, the assessment of the level of impact of elements measured by the survey is relatively consistent and similar among students enrolled in both French and English courses. However, the Program seems to have had a slightly more positive impact for the students in the English course on their use of the second language at work (6 points higher in the cumulative) and on their work performance (8 points higher in the cumulative).

An analysis on the level of impacts per participating organization reveals that the Program has about the same impact on students from MMS as those of other Sectors.

Table 3: Level of Impact of the Training Offered by the Program on the Students
  Cumul * Very
Positive		 Somewhat
Positive		 No Impact Somewhat
Negative		 Very
Negative		 No Response Total
Your ability to communicate in the second language for work purposes (speak, read, write, understand) 83 44.2% 45.3% 9.9%       181
On your level of confidence using the second language 79 35.4% 49.7% 13.3%     1.1% 181
On your use of the second language at work 74 26.0% 48.6% 23.8% 0.6%   0.6% 181
On your use of the second language outside of work 72 19.9% 48.1% 31.5%       181
On your performance 69 17.7% 44.8% 34.3% 0.6%   2.2% 181
On your interpersonal skills and your ability to work in a team 67 16.6% 39.8% 42.0% 0.0%   1.1% 181
On your morale at the office 72 28.2% 36.5% 33.1% 0.6% 1.7%   181
On your short-term objectives 74 26.5% 45.9% 26.0% 0.6% 1.1%   181
On your medium and long-term objectives 76 30.9% 45.9% 21.5% 1.1%     181
On your commitment to your employer 72 26.0% 39.2% 31.5% 1.7% 1.1% 0.6% 181
On your opinion of your employee's commitment to the development of his/her human resources 76 34.8% 39.8% 22.1% 1.1%   1.7% 181
On your awareness, appreciation and tolerance of linguistic duality in your workplace 73 31.5% 38.7% 24.3% 1.1%   3.9% 181
* The summary scale of impact results was calculated according to the averages found based on 5 points scoring scales used during the survey. The score would be 100 if all the respondents had indicated 'Very positive' and 0 if all the respondents had indicated 'Very negative'. Generally, a score lower than 70 indicates a relatively low positive impact and a score above 80 corresponds with a very positive impact. The entries are therefore weighted percentages where the value of 100 was attributed to 'Very positive', 75 for 'Somewhat positive', 50 for ' No impact', 25 for 'Somewhat negative' and 0 for 'Very negative'.

Source: Internet survey

Top of Page
6.5.2 Impact of the IHLTP on the Work Environment

According to the students, the training offered by the Program also had various effects on the work environment (Table 4). In particular, respondents indicated that the training has a positive impact on the importance that employers and employees give to language training and bilingualism. In fact, between 79% and 82% of respondents are very or somewhat positive about it. With regard to the level of impact the Program has had on leadership from directors in terms of bilingualism and organizational culture, respondents rated it as having a slightly less positive impact with 66% of respondents stating that the impact is positive.

However, when students assessed the impact of the training on the use of the second language in their work environment, they are generally much more harsh. 53% of respondents say that they have seen a positive impact on their use of the second language for communication in work meetings and in internal or external written communications. Moreover, only 17% of respondents note a very positive effect. This finding should be interpreted with caution since several respondents (30%) did not answer this question. These respondents answered the more specific questions on use of second language in the workplace rather than the overall survey questions. Still, the impact seems to be slightly more positive during verbal communication outside of formal meetings, with more than 60% of respondents contending that the training had a positive effect on their use of the second language in public areas and in internal verbal communication. Internal written communications and communication during work meetings do not seem, according to the students, to have been affected by the Program. Once again, the specific nature of the linguistic circumstances in each work environment may preclude the use of the second language during meetings.

Table 4: Student Opinion on the Program's Impact on their Branch's Work Environment
  Cumul * Very
Positive		 Somewhat
Positive		 No Impact Somewhat
Negative		 Very
Negative		 No
Response		 Total
On the importance given to language training and biligualism by employees 77 29.8% 52.5% 14.9% 0.6% 0.6% 1.7% 181
On the importance given to language training and biligualism by the employer 77 35.9% 43.1% 17.7% 0.6%   2.8% 181
On the leadership of directors regarding biligualism 70 22.1% 43.6% 29.3% 0.6% 1.1% 3.3% 181
On the organizational culture (way of thinking, managing and doing things) 68 19.3% 43.1% 31.5% 0.6% 0.6% 5.0% 181
On the use of the second language in your work environment 51 ** 16.6% 34.3% 18.2%     30.9% 181
During work meetings 65 13.8% 38.7% 43.6% 0.6% 0.6% 2.8% 181
In hallways or public areas 71 19.9% 48.6% 28.7%     2.8% 181
In written communications to people working internally 66 16.0% 34.8% 47.0%     2.2% 181
In internal verbal communications 69 17.1% 45.9% 34.3%   0.6% 2.2% 181
In written communications with people outside the work environment (clients and people from other departments) 65 16.6% 30.4% 51.4%     1.7% 181
On the use of the second language outside your workplace 67 16.6% 38.7% 42.5%     2.2% 181
* The summary scale of impact results was calculated according to the averages found based on 5 scoring scales used during the survey. The score would be 100 if all the respondents had indicated 'Very positive' and 0 if all the respondents had indicated 'Very negative'. Generally, a score lower than 70 indicates a relatively low positive impact and a score above 80 corresponds with a very positive impact. The entries are therefore weighted percentages where the value of 100 was attributed to 'Very positive', 75 for 'Somewhat positive', 50 for ' No impact', 25 for 'Somewhat negative' and 0 for 'Very negative'.
** Several survey respondents (30%) did not answer this question. These respondents responded to the survey questions which are more specific and not to the overall question on use of second language in the workplace.

Source: Internet survey

An analysis of survey results by Sector reveals that the Program has about the same impact on the work environments in MMS as it does in other Sectors. However, a slightly greater impact (3 points higher in the cumulative) was observed in MMS Branches regarding the use of the second language in the work environment and internal verbal communications. In particular, this analysis by Branch reveals that within the Energy Policy Sector the Program has only had a small impact on the use of the second language in public areas, in written communications with people outside the work environment (72%), in verbal communications (56%) and in internal written communications (44%). Seven students from the Communications Branch also have a neutral position regarding the Program's impact on the work environment. Four of them feel that the Program does not have any effect on their environment. This perception is shared by managers of this Branch who state that the Program did not have any effect of the use of the second language in the workplace.

The Program seems to have a more positive impact on the MMS-MTB-ODG where more than 40% of students stated that the Program had a very positive effect on their use of the second language at all levels. The situation is similar among students from the ETPS-CETC (Bell's Corners) were 59% stated that the Program had a very positive impact on their work environment.


Top of Page

6.5.2.1 Managerial Opinion on the Impact of the IHLTP Program on the
Work Environment

When the managers were questioned about the IHLTP's impact, they responded that the Program had created a very positive effect on the importance employers or the organisation give to language training and bilingualism (Table 5). Close to 67% of managers stated that this feature is very positive, while over 33% consider it to be somewhat positive. Also, almost 80% of managers feel that the Program had a very positive impact on accessibility to language training. All agree that the Program had positive effect on the image of language training and office morale. Another positive impact is that the training supported managerial leadership regarding bilingualism.

However, if the work environment is almost entirely Anglophone or completely Francophone, the Program has no impact on the internal development of bilingualism in the workplace because of the prevailing use of only one language.

Despite the fact that managers recognized several positive features of the Program's internal structure, managers do not perceive the Program as having a very significant effect on performance or employee commitment, organizational culture or use of the second language in the workplace in general. In fact, in their opinion, the Program's impact on second language use in their work environments seems moderate.

The Program seems to have a positive impact on the work environment. For example, managers believe it improves interaction between Francophone and Anglophone employees, employee moral and the work climate. As well, within some Branches, the use of both official languages has increased in some work environments.

Overall, based on the impact measured as part of the data collection for this evaluation, the results of this impact analysis suggest that the Program is successful, considering its mandate, objectives, and implementation, yet considering that the Program is relatively new, the IHLTP in itself does not seem to lead to cultural change or promote bilingualism in the workplace. In fact, prevailing levels and use of English and French appear to either promote or restrict the IHLTP's impact on the various work environments. Other collaborative efforts will have to be developed and implemented by senior management so that a cultural change towards bilingualism can take place.

Table 5: Opinions of Managers on the Program's Impact on the Work Environment at their Branch
  Cumul * Very
Positive		 Somewhat
Positive		 No Impact Somewhat
Negative		 Very
Negative		 No
Response		 n 
On the language skills of the staff at your Branch and Sector 86 44.4% 55.6% 0.0% 0.0% 0.0% 0.0% 9
On the accessibility of language training 94 77.8% 22.2% 0.0% 0.0% 0.0% 0.0% 9
On the immediate and constant application of and
learned skills in the workplace		 78 33.3% 44.4% 22.2% 0.0% 0.0% 0.0% 9
On emplyee performance 64 22.2% 33.3% 33.3% 0.0% 0.0% 11.1% 9
On office morale 83 33.3% 66.7% 0.0% 0.0% 0.0% 0.0% 9
On employee commitment to your organiztation 72 11.1% 66.7% 22.2% 0.0% 0.0% 0.0% 9
On the importance given by employees to
language training and biligualism		 86 44.4% 55.6% 0.0% 0.0% 0.0% 0.0% 9
On the importance given by employers or
your organization to language training and biligualism		 92 66.7% 33.3% 0.0% 0.0% 0.0% 0.0% 9
On your leadership concerning biligualism 89 55.6% 44.4% 0.0% 0.0% 0.0% 0.0% 9
On the organizational culture (way of thinking, managing
and doing things)		 78 22.2% 66.7% 11.1% 0.0% 0.0% 0.0% 9
On the image of language training 89 55.6% 44.4% 0.0% 0.0% 0.0% 0.0% 9
On the awareness, appreciation and tolerance of
linguistic duality in your workplace		 75 44.4% 11.1% 44.4% 0.0% 0.0% 0.0% 9
On the use of the second language in your workplace
in general and specifically		 69 0.0% 77.8% 22.2% 0.0% 0.0% 0.0% 9
At work meetings 67 11.1% 44.4% 44.4% 0.0% 0.0% 0.0% 9
In hallways or public areas 67 11.1% 44.4% 44.4% 0.0% 0.0% 0.0% 9
In written communications sent to others internally 58 0.0% 33.3% 66.7% 0.0% 0.0% 0.0% 9
In verbal communications taking place internally 75 11.1% 77.8% 11.1% 0.0% 0.0% 0.0% 9
In written communications sent to individuals outside the
work environment (clients and people from other Departments)		 67 11.1% 44.4% 44.4% 0.0% 0.0% 0.0% 9
In verbal communications with people outside the workplace (clients
and people from other Departments)		 69 11.1% 55.6% 33.3% 0.0% 0.0% 0.0% 9
On the number of employees able to fill positions at higher levels 75 11.1% 77.8% 11.1% 0.0% 0.0% 0.0% 9
On the decrease in time required for full-time language training 75 44.4% 11.1% 44.4% 0.0% 0.0% 0.0% 9
* The summation of the impact results was calculated according to the averages found based on 5 scoring scales used during the survey. The score would be 100 if all the respondents had indicated 'Very positive' and 0 if all the respondents had indicated 'Very negative'. Generally, a score lower than 70 indicates a relatively low positive impact and a score above 80 corresponds with a very positive impact. The entries are therefore weighted percentages where the value of 100 was attributed to 'Very positive', 75 for 'Somewhat positive', 50 for ' No impact', 25 for 'Somewhat negative' and 0 for 'Very negative'.

Source: Telephone interviews with managers

Top of Page
6.6 Conclusion

The overall evaluation of the IHLTP by students and key stakeholders is very positive. The language training program is highly regarded by all of the stakeholders questioned who showed a very high level of satisfaction. The Program addresses the specific needs of both employees and managers, and responds to a relatively significant (a 30% participation rate on average) and constant demand. Furthermore, the Program seems unique and complements other types of training available to Departmental employees.

Although the specific objectives of the IHLTP do not include helping students attain language skill levels and passing the Public Service Commission (PSC) exams, a high number of employees and managers contend that the Program should include services involving courses aimed at helping employees pass these exams. Moreover, some employees feel that the course flow is too slow and the frequency is not sufficient for meeting such objectives. Preliminary data provided by the Program's management suggests, however, that the Program includes some activities toward achieving PSC exams (for e.g., prepartation for the written exams) and that teachers spend approximately 120 hours per session outside of class time in private lessons with employees, a majority of whom are in the process of advancing to another PSC language skill level.

The Evaluation Division was not able to obtain, nor analyse detailed data on individual student progress during the evaluation. This is due in part to unavailability of comprehensive and consistent electronic data and the difficulties encountered in comparing IHLTP language skill levels to those of other classification system (e.g., the Public Service Commission (PSC) scale for comprehension, writing and oral interaction). The Program would benefit from ensuring that student progress is gathered, analysed and made available. For example, student progress could be measured by using the PSC classification system as a refence point (students could use online practice tests or get tested throught PSC exams).

A strategy for the development and expansion of the IHLTP would be beneficial in order to be able to offer new services to participating Branches or to expand the Program to new Branches. A complete assessment process could be undertaken to determine whether the need for language training services in other sectors and branches exists in order to establish the requirements, objectives and expectations of potential students and management. This needs assessment could also include the personal needs of employees as well as organizational needs.

Lastly, several respondents fear that expanding the Program to a broader audience may lead to a centralization of the Program's management and a decrease in training quality. Nevertheless, considering the language training Program's success, the high demand, the needs met by the Program and its positive impact on employees and the work environment, there is no doubt about the value of supporting this Program and in providing sufficient resources to allow it to develop, and maintain its level of operation and its quality of service.


Top of Page

7.0 RECOMMENDATIONS AND MANAGEMENT
RESPONSES

Recommendation #1: Review the appropriateness of requiring participants to make up a portion of the hours spent in training

The existing model is ineffective. It is recommended that the ADM of the Minerals and Metals Sector ensure that a strategy is established to determine the best approach. If the existing model is the most appropriate, its foundation, application and monitoring should be reviewed. Furthermore, if a new model is chosen, an implementation and monitoring strategy should be developed.

Management Response

Accepted. The executive committee of the MMS, as well as the consultative committee on non-statutory language training, and the participating managers will be consulted to examine the shortcomings of the existing model and determine how to make it more efficient and equitable.

Date of completion:

Review of the model: June to December 2006.

Development of a revised policy: January 2007.

Evaluation of the changes: June 2007.

Recommendation #2: Collecting and capturing key information

It is recommended that the ADM of the Minerals and Metals Sector ensure that the In-House Language Training Program implements a strategy for collecting and capturing key information on the Program's activities as well as disseminating information to managers and students.

Management Response

Accepted. Certain mechanisms for collecting data have already been initiated through the evaluation process. A new database is currently being developed to allow for easier access and more complete information. Also, the Program will establish performance indicators and disseminate information to the managers and students.

Date of completion : March 2007.

Furthermore, the program will be taking part in a research project entitled Professional bilingualism and biliteracy: The contexts and practices of bilingual communication in the Canadian public service, being done by Carleton University. The study will follow a group of NRCan employees, some in training with the In-House program and others on training outside the department, for a year. The results of the study will provide further information on the program's impact in the work place.

Recommendation #3: Communication strategy

It is recommended that the ADM of the Minerals and Metals Sector ensure that the IHLTP develop a communication strategy to promote the Program.

Management Response

Accepted. A business relations and communication strategy will be developed, which will include a stronger presence on management committees of participating branches and implementing an advisory board that includes representatives of sectors and branches that use the MMS training Program's services. As well, the means for improving the dissemination of information to employees in general will be examined.

Date of completion: December 2006.

1 See: Follow-up Audit of Official Languages - Language of Work (A04014)
2 Margin of error of 2.50%, confidence interval at 95%.
3 With a confidence interval of 95% and a margin of error of 3.78%.
4 With a margin of error of 1.4% for a confidence interval of 99%.
5With a margin of error of 1.4% for a confidence interval of 99%.
6 Respective margins of error of 3.71% and 4.73% for a confidence interval of 95%.