3.2 The Context of Evaluation
According to Tomlinson (2003c), Materials evaluation is a procedure that involves measuring the value (or potential value) of a set learning materials.
It is probably reasonable to assume that there are very few teachers who do not use published course materials at some stage in their teaching career.
At this stage, a distinction should be made between teaching situations where open-market materials are chosen on the one hand, and where a Ministry of Education (or some similar body) produces materials that are subsequently passed on to the teacher for class room use on the other.
Here we have two context: the one which promotes open-market materials, and the one which imposes the materials prepared/suggested by Ministry of Education.
The nature of the evaluation process in each of these scenarios will probably differ as well.
In the first type of situation teachers may have quite a large amount of choice in the materials they select, perhaps being able to contact really with colleagues and a Director with respect to this material.
However, the second situation will more than likely involve teachers in an understanding of why the materials have been written in a such a way and how they can make effective use of them in the classroom.
The second situation is more restricted for the teachers, but might motivate the teachers to more evaluate their materials under limited circumstances.
For the vast majority of teachers working in the first situation, that of having a good deal of choice in the selection appropriate materials, writing their own materials can be very time consuming and not necessarily cost-effective.
Today there is a wealth of EFL material available, with literally hundreds of new, commercially available titles appearing every year in English-speaking countries.
Wider choice means more need for evaluation prior to selection. And, in response to such a demand, there are some journals which have regular reviews of recently published materials.
Moreover, teachers or course organizers are often under considerable professional and financial pressure to select a coursebook for an ELT programme that will then become the textbook for years to come.
No matter in which context a teacher might be working, no textbook or set of materials is likely to be perfect, and there does not seem as yet an agreed set of criteria or procedures for evaluation.
“This is inevitable as the needs, objectives, backgrounds and preferred styles of the participants differ from context to context” (Tomlinson, 2003c: 15).
Nevertheless, there is always a need some mode for hard-pressed teachers or course planners that will brief, practical to use and yet comprehensive in its coverage of criteria.
Therefore, this chapter offers a model which distinguishes the purpose behind the evaluation and which consists of two stages; an external evaluation that offers a brief overview of the materials from the outside (cover, introduction, table of contents), which is then followed by a closed and more detailed internal evaluation.
Of course the evaluation process is
never static; when materials are deemed appropriate for a particular course after a preliminary evaluation, their ultimate success or failure may only be determined after a certain amount of classroom use (while- and post-use evaluation).
The External Evaluation
In this central stage of the mode, the authors included criteria that will provide a comprehensive, external overview of how the materials have been organized.
The aim is basically that of examining the organization of the materials as stated explicitly by the author/ publish by looking at:
- the blurb, or the claims made on the over teacher’s/students’ book.
- the introduction and table of contents
We also find it useful to scan the table of contents page int hat it often represents a bridge between the external claim made for the materials and what will actually be presented in the materials themselves.
At this stage we need to consider why the materials have been produced. Presumably because the author/publisher feels that there is a gap in the existing market that these materials are intended to fill: so we shall have to investigate this further to whether the objectives haven clearly spelled out.
Here is an example of one such blurb taken from a well-known EFL textbook published in 2012:
and integrated skills series which is designed to offer flexibility with different teaching and learning styles. Fun for learners to use and easy for teacher to adapt.
- Fully integrated grammar, skills and lexical syllabuses provide a balanced learning experience
- Engaging topics motivate students and offer greater personalization
- A wide range of approaches exploit different learning styles
- Clearly structured grammar presentations are reinforced with extensive practice
- A variety of listening and speaking activities develop learning fluency
- Learner training throughout the Student’s Book and Workbook maximizes skills development
What can we understand from these information in terms of external evaluation?
It appears that this textbook is aimed at intermediate level students with different learning styles and different levels of motivation who will benefit from learner training.
This textbook also seems to be designed for flexible use and to offer and integrated learning experience covering grammar, lexis and skills.
Later, when the evaluator investigates the organization of the materials s/he will have to ascertain whether or not this is really the case.
The following example example is part of the introduction take from a recent EFL series. Certain terms and key concepts that might need further investigation area italicized.
Tasks and activities are designed to have a
real communicative purposerather than simply being an excuse to practice specific features.
We have placed a special emphasis on representing
an accurate multicultural view of English as it is spoken today.Many courses still represent the English-speaking world as being largely UK- and US-based. considering the fact that there are now more non-native English speakers than native, we have also included a variety of accents from a wide range of countries and cultures.
Throughout the Student’s Book,
learner autonomyis promoted via clear cross-referencing to features in the Workbook and elsewhere. Here students can find all the help and extra practice they need.
Just to remind it again:
During the external evaluation stage we should examine (1) the claims made for the materials by the author/publisher with respect to the intended audience, (2) the proficiency level, the context and presentation of language items, (3) whether the materials are to be core or supplementary, (3) the role and availability of a teacher’s book, (5) the inclusion of vocabulary list/index, (6) that table of contents, (7) the use of visuals and presentation, (8) the cultural specificity of the materials, (9) the provision of digital materials and (10) inclusion of tests.
3.4 The Internal Evaluation
We now continue to the next stage of our evaluation procedure by performing an in-depth investigation into the materials.
The essential issue at this stage is for us to analyze the extent to which the aforementioned factors in the external evaluation stage match up with the internal consistency and organization of the materials as stated by the author/publisher.
In order to perform an effective internal inspection of the materials, we need to examine at least two units (preferably more) of a book or set of materials to investigate the following factors:
The presentation of the skills in the materials.
The grading and sequencing of the materials
Where reading/discourse skills are involved, is there much in the way of appropriate text beyond the sentence?
Where listening skills are involved, are recordings authentic or artificial?
Do speaking materials incorporate what we know about the nature of real interaction or are artificial dialogues offered instead?
The relationship of tests and exercises to (1) learner needs and (2) what is taught by the course material.
Do you feel that the material is suitable for different learning styles? Is a claim an provision made for self-study and is such a claim justified?
Are the materials engaging to motivate both students and teachers alike, or would you foresee a student/teacher mismatch?
Just to remind it again:
During the internal evaluation stage, we should examine (1) the treatment and presentation of the skills, (2) the sequencing and grading of the materials, (3) the type of reading, listening, speaking and writing materials contained in the materials, (4) appropriacy of tests and exercises, (5) self-study provision and (6) teacher-learner balance in use of the materials.
3.5 The Overall Evaluation
At this stage we may now make an overall assessment as to the suitability of the materials by considering the following parameters:
The usability factor. How far the materials could be integrated in to a particular syllabus as core or supplementary.
The generalizability factor. Is there a restricted use of core features that make the materials more generally useful?
The adaptability factor. Can parts be added/extracted/used in another context/modified for local circumstances?
The flexibility factor. How rigid is the sequencing and grading? Can the materials be entered at different points or used in different ways?
Conclusion: The Flow of Evaluation
macro-evaluation (External) -> Inappropriate OR potentially appropriate | | exit | micro-evaluation (Internal) <---------------------------------/ | / \ inappropriate appropriate --> Adopt (make changes if needed)/select | exit