MATERIAL EVALUATION

MATERIAL EVALUATION

ido setyawan 1211010083

LECTURER : Eva Nurchurifiani, M.Pd.

THE STATE INSTITUTE OF ISLAMIC STUDIES

RADEN INTAN LAMPUNG

2013-2014

FOREWORD

Alhamdulillahirobbil’alamiin

The authors would like extend their very gratitude to Allah SWT, the Almighty, for the unlimited  blessings bestowed upon them one of wich is their great chance to accomplish composing this papers. Who has given affection for the author for taking the time to complete this papers titled“From Syllabus Design to Curriculum Development”.

It is highly expected that this papers might contribute to the betterment of English intruction in this institution. The authors are aware that this papers is still far from perfect. Therefore, the authors expect criticism and suggestions either in writing or orally.

Bandar Lampung, ............. March 2014

     The Authors

CHAPTER  I

BACKGROUND

A. Introduction

  Curriculum is the complete set of taught material in a school system. It is prescriptive (as opposed to the ‘descriptive’ syllabus, which is the outline of topics covered. If the curriculum prescribes the objectives of the system, the syllabus describes the means to achieve them).Curriculum comes from a Latin word which means the course of a chariot race.

However, curriculum has come to mean much more than a prescribed one track race and calls for a search for an understanding that gives meaning to education that is both functional and ethicalCurriculum as a guiding document helps teachers in understanding standards that students need to achieve at the end of a developmental stage.The curriculum document will indicate “what” to teach, ”how” the curriculum is to be taught and help in checking “whether” the curriculum is taught as per the document.

Over the years, ‘curriculum’ has meant different things to different educationists. Some simply equate curriculum to the syllabus that is to be transmitted in the class. “A syllabus gives a more focused outline for particular subjects. It can’t be equated, because a curriculum is for a course but a syllabus is for a subject,” says Dr. Yasmin Jayathritha. The curriculum is the superset and syllabus is the subset of curriculum. The syllabus is the content, the list of topics/concepts to be taught, whereas the curriculum is a consideration of the objectives, the content, methods chosen to achieve those objectives

CHAPTER II

Materials Evaluation

1.      What is Materials Evaluation?

Materials Evaluation is a procedure that involves measuring the volue ( or potencial volue) of a set of  learning materials. It involves making  judgements about  the effect of the materials on the people using them and it tries to measure some or all of the following.

·         The appeal of the materials to the learners;

·         He credibility of the materials to learners, teachers and administrator;

·         The validity pf materials (i.e., is what they teach wotrh teaching?);

·         The reability of the materials (i.e.,would they have the same effect with different groups of s of the target learners?);

·         T he ability of the materials to interest the learners and the teacher;

·         The ability of the materials to motivate the learners;

·         The value of the materials in the terms and short-term learning  (important, for example, for performance on test and examinations);

·         The value of the materials in terms of long-term learning (of both language and of communication skills);

·         The learners’  perceptions of the value of the materials;

·         The teachers’ percepions of the value of the materials;

·         The assistance given to the teachers in terms of preparation, dilivery and assesment;

·         The flexibility of the material (e.g., the extent to which it is easy for a teacher to adapt the materials to suit a particular context);

·         The contribution made by the  materials to the teacher development;

·         The macth with administrative requirements(e.g., standardization across classes, coverage of syllabus, preparation for an examination).

It advious from a consideration of the effect above that no two evaluations can be the same, as the objectives, backgrounds and preferred styles of participans will be differ  from context to context.

An evaluation is not same as an analysis. It can include an analysis or follow from one, but the objectives and procedure are different. An evaluations focuses on the users of the materials and makes judgements about their effects. No matter  how stuctured, criterion referenced  and rigorous  an evaluation is, it be essentially subjective. On, the other hand, an analysis focuses an on the materials and it aims to provide an objective analysis thet.

It can also be given a numerical valueor and after many such questions have been asked about the materials, subtotal scores and total scores can be calculated and indications can be derived of the potential  value of the materials and subsections of them.

            A detailed analysis of a set of materials can be very useful for deciding. Many publications on matrials evaluation mix analysis and evaluation and make it verydifficult to use their suggested criteria. My preference for separating analysis from evaluation is shared by little john (1998) who parents a general framework for analysis materials(pp.192-22) , which he suggests could be used prior to evaluation and action in a model which is sequenced as follow:

·         Analysis of the target situation of use.

·         Material analysis.

·         Macth and evaluation (determining the approciacy of the materials to the target situation of use).

·         Action.

2.      Pinciple in Materials Evaluation

Many evaluations are impressionistic, or at best are aided by an ad hoc and very subjective list of criteria. In my view it is very important that evaluations (even the most informal ones) are driven by a set of principle and that these principles are articulated by evaluator (s)prior  to the evaluation. In this way greater validity and realibility can be achieved and fewer mistake are likelyto be made.

            In developing a set of principles itis useful to consider  the folling:

                                    The Evaluator’s Theory of Learning and Teaching

All teachers develop theories of learning and teaching which they apply in their classrooms (even though they are often unawer of doing so ). Many researchers (e.g., Schon, 1983) argue that it is useful for teachers to try to achieve an articulation of their theories by reflecting on their practice. For example Edge and wharton  (1998: 297) argue that reflective practice can not only lead to perceived improvements in practice but, more importantly, to deeper understandings of area investigated. At the same time evaluators can learn a lot about themselves and about the learning and teaching process.

            Here are some of my theories, which I have articulated as a result of reflection on my own and other teachers’ practice:

·         Language learners succeed best if learning is a positif, relaxed and enjoyable experience.

·         Language teachers tend to teach most succesfully if they enjoy their role and if they can gain some enjoyment themselves from the materials they are using.

·         Learning materials lose credibility for learners if they suspect that the teacher does not value them.

·         Each learner is different from all the others in a class in terms of his or her personality, motivation, attitude, aptitude, prior experience, interests, needs, wants and preferred learning style.

·         Each learner varies from day to day in terms of motivation, attitude, mood, percieved needs and wants,enthusiasm and energy.

·         There are superficial culturall differences between learners from diferent coutries but there are also srtong universal determinants of successful language teaching and learning.

·         Succesful language learning in classroom depends on the generation and maintenance of high levels of energy.

·         The teacher is respinsible for the initial generation of energy in a lesson; good materials can thenn maintain andeven increase that energy.

·         Learners only learn what they really need or want to learn.

3.      Learning Theory

Research into learning is controversial as there are so many variables involved and local circumtances often make generalization precarious. However, it is important that the materials evaluator considers the findings of learning research and decides which of its findings are convincing and applicable. The conclusions which convince me are that:

·         Deep processing of intake is required if effective and durable learning is to take place.

·         Affective engagement is also essential for effective and durable learning.

·         Making mental connections is a crucial aspect of the learning process.

·         Experiantial learning is es, assential and, in particular, apprehension should come to the learner before comprehension.

·         Learners will only learn if they need and want to learn and if they are willing to invest time and energy in the process.

·         Materials which address the learner in an informal, personal voice are more likely to facilitate learning than those which use a distant, formal voice (Beck etal., 1995;Tomlison, 2001b).

·         Multidimensional processing of intake is or essential for succesful learning and involves the learner creating a mental representation of the intake through such mental processes as sensory imaging, effective association and the use of the inner voice (Kaufan, 1996;Masuhara, 1998a; Tomlinson, 2000b, 2000c, 2001a ).

4.      Second language acquisition Research (SLA)

SLA research is so far inconclusive and has stimulated many disagreements and debates. However, there is now a sufficient consensus of opinion on certain facilitating features of language learning for them to be useful in helping to articulate the principles to be used as a basis of the materials evaluation. In Tomlinson (1998b: 5-22) I discussed the principles of second language acquisition which I think SLA researchers would  agree are relevant to the development of materials for the teaching of languages. Some of these principles are summarized below:

·         Materials should achieve impact.

·         Materials should help learners to feel at case.

·         Materials should help the learners to develop confidence.

·         What is being taught should be perceived by learners as relevant and useful.

·         Materials should require and facilitate learner self-investment.

·         Learners must be ready to acquire the points being taught.

·         Materials should expose the learners to language in authentic use.

·         The learners’ attention should be drawn to linguistic features the input.

·         Materials should provide the learners with opportunities to use the target language to achieve communicative purpose.

·         Materials should take into account that the positive effect of instruction are usually delayed.

·         Materials should take into account that learners differ in learner styles.

·         Materials should into account  that learners differ in affective attitudes.

·         Materials should maximize learner potential by encouraging intellectual, aesthetic and emotional involvement which stimulates both right and left brain activities.

·         Materials should provide opportunities for outcome feedback.

Richards (2001: 206) considers this ‘list’ to be somewhat cumbersome..to be apply’ and he suggests the following list of the ‘qualities each unit in the materials should reflect’:

·         Give learners something they can take away from the lesson.

·         Teaches something learners feel they can use.

·         Gives learners a sense of achievement.

·         Practices learning items in an interesting and novel way.

·         Provides a pleasurable learning experience.

·         Provides opportunities for individual practice.

·         Provides opportunities for personalization.

·         Provides opportunities for self-assessment of learning.

At the risk of becoming even more cumbersome, I would now add the fallowing to my list:

·         Materials should help the learner to develop cultural awareness and sensitivity.

·         Materials should  reflect the reality of language use.

·         Materials should help learners to learn in ways similar to the circumstance in which they will have to use the language.

·         Materials should help to create readiness to learn.

·         Materials should achieve affective engagement.

The important thing is for materials evaluators to decide for themselves which findings of SLA research they will use to develop principles for their evaluation. Ultimately what matters is that an evaluation is principled, that the evaluator’s principles are made overt and that they are referred to when determining and carrying out the procedures the evaluation.

5.      Types of Materials Evaluation

There are many different types of materials evaluation. It is possible to apply the basic principles of materials evaluation to all types of evaluation but it is not possible to make generalizations about procedures which apply to all types. Evaluations differ, for example, in purpose, in personnel, in formality and in timing.

6.      Pre-use Evaluation

Pre-use evaluation involves making prediction about the potential value of materials for their users. It can be context-free, as in are view of materials for a journal, context-influenced as in a review of draft materials for a publisher with target users in mind or context-dependent, as when a teacher select a course-book for use with her particular class. Often pre-use evaluation is impressionistic and consist of a teacher flicking through a book to again a quick impression of its potential value. This is especially true if more than two evaluators conduct the evaluation independently and then average their conclusions. For example, in the review of eight adult EFL courses conducted by Tomlinson et al.(2001) , the four evaluators devised 133 criteria together and then used them independently and in isolation to evaluate the eight course before pooling their data and averaging their scores. Even then, though, the reviewers admitted that, the same review, conducted by a different team of reviewers, would almost certainly have produced a different set of results’(p. 82).

7.      Whilst-use Evaluation

This involves measuring the value of materials whilst using them or whilst observing them being used. It can be more objective and reliable than pre-use evaluation as it makes use of measurement rather than  prediction. However, it is limited to measuring what is observable (e.g. are the instructions clear to the learners?) and cannot claim to measure what is happening in the learners brains. It can measure short-term memory through observing learner performance on exercises but it cannot measure durable and effective learning because of the delayed effect of instruction. It is therefore very useful but dangerous too, as teacher and observers can be misled by whether the activities seem to work or not.

            Exactly what can be measured in a whilst-use evaluation is controversial but I would include the following:

·         Clarity of instructions

·         Clarity of layout

·         Comprehensibility of texts

·         Credibility of tasks

·         Achievability of tasks

·         Achievement of performance objectives

·         Potential for localization

·         Practicality of the materials

·         Teachability of the materials

·         Flexibility of the material

·         Appeal of the material

·         Motivating power  of the material

·         Impact of the material

·         Effectiveness in facilitating short-term learning

Most of the above can be estimated  during an open-ended, impressionistic observation of materials in use but greater reliability can be achieved by focusing on one criterion at a time and by using pre-prepared instruments of measurement. For example, oral participation in an activity can be measured by recording the incidence and durationof each student’s oral contribution, potential for localization can be estimatedby nothing the times the times the teacher or student refers to the localization of learning while using the materials and even motivation can be estimated by nothing such features as student eye focus, proximity to the materials, time on task and facial animation.

Whilst-use evaluation receives very little attention in the  literature, but Jolly and Bolito (1998) describe interesting case studies of how student comment and feedback during lessons provided useful  evaluation of materials, which led to improvements being made in the materials during and after the lessons.

8.      Post-use Evaluation

Post-use evaluation is probably the most valuable (but least administered) type of evaluation as it can measure the actual effects of the materials on the users. It can measure the short-term effect as regards motivation, impact, achievability, instant learning, etc., and it can measure the long-term effect as regards durable learning and application. It can answer such important questions as:

·         What do the learners know which they did not knowbefore starting to use the materials?

·         What do the learners still not know despite using the matials?

·         What can the learners do which they could not do before starting to use the materials?

·         What can the learners still not do despite using the materials?

·         To what extent have the materials prepared the learners for their examinations?

In other word, it can measure the actual outcomes of the use of the materials and thus provide the data on which reliable decisions about the use, adaptationor replacement of the materials can be made.

Ways of measuring the post-use effects of materials include:

·         Tests of what has been ‘taught’ by the materials

·         Tests of what the students can do

·         Examinations

·         Interviews

·         Questionnaires

·         Criterionreferenced evaluations by the users

·         Post-course diaries

·         Post-course ‘shadowing’of the learners

·         Post-course reports on the learners by employers, subject tutors,etc.

The main problem, of course, is that it takes time and expertise to measure post-use effects reliably (especially as, to be really revealing, there should be measurement of pre-use attitudes and abilities in order to provide data for post-use comparison). But publishers and ministries do have the time and can engage the expertise, and teachers can be helped to design, administer and analyse post-use instruments of measurement.  Then we will have much more useful information, not only about the effects  of particular courses of materialsbut about the relative effectiveness of different types of materials. Even then, though, we will need to be coutions, as it will be very difficult to separate such variables as teacher effectiveness, parental support, language exposure outside the classroom, intrinsic motivation, etc.

9.      Developing criteria for materials evaluation

One way of developing a set of criteria is as follows:

1.      Brainstorm a list of universal criteria

Universal criteria are those which would apply to any language learning materials anywhere for any learners. So, for example, they would apply equally to a video course for ten-year-olds in Argentina and an English for academic purpose textbook for undergraduates in Thailand. They derive from principles of language learning and provide the fundamental basis for any  materials evaluation. Brainstorming a random list of such criteria (ideally with other colleagues) is a very useful way of beginning an evaluation, and the most useful way i have found of doing it is to phrase the criteria as specific questions rather than to list them as general headings.

Example of universal criteria would be:

Are the instructions clear?

Are the materials likely to achieve affective engagement?

2.      Subdivide some of the criteria

If the evaluation is going to be used as a basisfor revision or adaptation of the materials, or if it is going to be  a formal evaluation and is going to inform important decision, it is useful to subdivide some of the criteria into more specific questions.

For example:

Sufficient?

Separated?

Such a subdivision can help to pinpoint specific aspects of the materials which could gain from revision or adaptation.

3.      Monitor and revise the list of universal criteria

Monitor the list and rewrite it according to the following criteria:

Is each question an evaluation question?

If a question is an analysis question (e.g.,’Does each unit include a test?’) then you can only give the answer a 1 or a 5 on the five-point scale which is recommended later in this suggested procedure.

Does each question only ask one question?

Many criteria in published lists ask two or more  questions and therefore cannot be used in any numerical grading of the materials. For example, includes the following question which could be answered ‘Yes,No,’ or ‘No, Yes’.

There are a number of ways in which each question could be rewritten to make it more reliable and useful. For example:

Are the communicative tasks useful in providing learning opportunities for the learners?

Are the activities in each unit linked to each other in ways in help the learners?

4.      Categorize the list

It is very useful to rearrange the random list of universal criteria into catergories which facilitate focus and enable generalizations to be made. An extra advantage of doing this is that you often think of other  criteria related to the category as you are doing the categorization exercise.

Possible caegories for universal criteria would be:

Learning principles

Cultural perspective

Topic content

Teaching points

Texts

Activities

Methodology

Instructions

Design

5.      Develop media-specific criteria

These are criteria which ask questions of particular relevance to the medium used by the materials being evaluated (e.g.,criteria for books, for audio cassettes, for videos, etc.) examples of such criteria would be:

Is it clear which sections the visuals refer to?

Are the different voices easily distinguished?

Obviously these criteria can also be usefully categorized (e.g.,under illustrations,layout, audibility, movement).

6.      Develop content-specific criteria

These are criteria which relate to the topics and/or teaching points of the materials being evaluated. Thus there would be a set of topic related criteria which would be relevant to the evaluation of a business English textbook but not to a general English coursebook, and there would be a set of criteria relevant to a reading skills book which would not be relevant to the evaluation of a grammar practice book and vise versa. Examples of content-specific criteria would be:

Do the examples of business texts (e.g.,letters,invoice,etc.) replicate features of real-life business practice?

7.      Develop age-specific criteria

These are criteria which relate to the age of the target learners. Thus there would be criteria which are only suitable for five-years-old, for ten-year-old, for teen-agers, for young adults and for mature adult. These criteria would relate to cognitive and affective development, to previous experience, to interests and to wants and needs.

Examples of age-specific criteria would be:

Are there short, varied activities which are likely to match the attention span of the learners?

Is the content likely to be cognitively challenging?

8.      Develop local criteria

These are criteria which relate to the actual or potential environment of use. They are questions which are not concerned with estabilishing the value of the materials perse but rather with measuring the value of the materials for particular learners in particular circumstances. It is this set of criteria which is unique to the specific evaluation being undertaken and which is ultimately responsible for most of the decisions made in relation to the adoption, revision or adaptation of the materials.

Typical features of the environment which would determine this set of materials are:

·         The type(s) of institution(s)

·         The resources of institution(s)

·         Class size

·         The background,needs and wants of the learners

·         The background,needs and wants of the teacher

·         The language policy in operation

·         The syllabus

·         The objectives of the courses

·         The intensity and extent of the teaching time available

·         The target examinations

·         The amount of exposure to the target language outside the classroom.

9.      Develop other criteria

Other criteria which it might be appropiate to develop could include  teacher-specific, administrator-specific, gender-specific, culture specific, especially in the case of review for a journal, criteria assessing the match between the materials and claims made by the publishers for them.

10.  Trial the criteria

It is important to trial the criteria (even prior to a small, fairly informal evaluation) to ensure that the criteria are sufficient, answerable, reliable, and useful. Revisions can then be made before the actual evaluation begins.

11.  Conducting the evaluation

From experience, i have found the most effective way of conducting an evaluation is to:

·         Make sure that there is more than one evaluator

·         Discuss the criteria to make sure there is equivalence of interpretation

·         Answer the criteria independenly and in isolation from the other evaluator(s)

·         Focus in a large evaluation on typical unit for each level (and then check its typicality by reference to other units)

·         Give a score for each criterion (with some setsof criteria weighted more heavily than others)

·         Write comment at the end of each category

·         At the end of the evaluation aggregate each evaluator’s scores for each criterion, category of criteria and then average the scores

·         Record the comments shared by the evaluators

·         Write a joint report.

CHAPTER III

CONCLUSION

Materials evaluation is initially a time-consuming and difficult undertaking. Approaching it in the principled, systematic and rigorous ways suggested above cannot only help to make and record vital discoveries about the materials being evaluated but can also help the evaluators to learn about the materials, aboutlearning and teaching and about themselves. In addition, doing evaluations formally and rigorously can eventually contribute to the development of an ability to conduct principled informal evaluations quickly and effectively when the occasion demands (e.g.,when asked for an opinion of a new book, when deciding which materials to buy in a bookshop, when editing oyher people’s materials). I have found evaluation demanding  but rewarding. Certainly, i have learned a lot every time i have evaluated materials, wether it be the worldwide evaluation of a course book i once undertook for a British publisher , the evaluation of computer software i once undertook for an American company, the evaluation of materials i have done for reviews in ELT Journal or just looking through materials in a bookshop every time i visit my family in cambridge. I hope, above all else, that i have learned to be more open-minded and that i have learned what criteria i need to satisty when i write the best-selling coursebook i still plan one day to write.

REFERENCE

Tomlinson,Brian,2007,Deloping Material for Language Teaching,New York:British Library Cataloguing in Publication Data.