 |
 |
 |
|
 |
|
 |
|
Making it real? Notes
on the sample materials for the
National tests in which the sole outcomes are overall levels are simply rough sorting procedures with no particular pedagogic functions. The aims are to place pupils on national scales of achievement in defined areas of the curriculum, 'subjects'. It is not the purpose of these notes to discuss the particular changes to the KS3 English tests in 2003, but simply to review the sample test materials in these terms, the key questions being those of validity and reliability:- · What
do the tests purport to assess? A set of tests with overlapping objectives is projected for this overall sorting procedure:- From 2003 separate levels will be awarded for Reading and Writing. The individual Reading and Writing mark schemes are not level related. The Reading level will be awarded on the basis of an aggregation of the marks achieved on the Reading paper and the Shakespeare reading task. Pupil performance across the two tasks may vary and the marking criteria are designed to recognise and reward a range of qualities in each. Similarly, the Writing level will be awarded on the basis of an aggregation of the marks achieved on the Writing paper (longer writing task) and the Shakespeare writing (shorter writing task) and pupil performance across the two tasks may vary. Pupils will also receive an overall English level on the basis of the aggregation of the total marks for Reading and the total marks for Writing. - with sets of assessment focuses (AFs) variously ascribed. In the Shakespeare paper the 'Writing' tasks have only a tangential connection with the plays studied. With some fine scoring of particular points in the tasks set, this seems on the face of it a somewhat elaborate procedure for the simple reporting in overall levels required - with reliability at risk. Unless marks are awarded across the whole range of broad mark bands there can be a cumulative regression of marks to the mean in the aggregations of question, paper and subject marks resulting in a consequent loss of overall discrimination and the maximum numbers of pupils at risk at the level boundaries, especially in the mid ranges in over-peaked curves of distribution. (The ideal 'bell curve' of distribution would be a horizontal straight line, with pupils evenly spread across the whole aggregated mark ranges and so relatively few at risk at the level boundaries.)
'Making it Real':- The test is based on a reading booklet which includes three texts, covering a range of genres and styles, literary and non-literary. - and apparently linked thematically:- Many people, from the ingenious scientist to the creative artist, are fascinated by the thought of re-creating a human being. Puppets, waxwork models, robots and computer animation are all attempts to do this in different ways and for different ways - except that only one question (14) appears to make any connections at all, and that for but two of the 32 marks, and for but two of the three passages. This proves spurious in any case:- Complete the table below, suggesting · one
purpose of each text - in which no link is actually made between the two texts. So, this apparently attractive contextualisation of the three passages is entirely spurious as far as the set tasks are concerned; they are each considered quite independently of each other. Where is the prompt to consider the texts in relation to each other and write on some aspect of the outlined linking theme that might have been looked for in a concluding, 'synoptic' question? But this is 'Reading', and the assessment focuses forbid:- Each question has an assessment focus which indicates the aspect of reading being assessed. So this final question has a degree of triviality at the highest level (AF6: 'writers' purposes and viewpoints, and the effect of the text') matched only by that of the first question: 'one purpose…', 'one word or phrase to describe the language used' - for each text and for but one mark each? The mark scheme lists contrasting one-word 'correct answers', as if anything half relevant will do (but 'Do not accept descriptive') and no further reflection is required. In these pre-tested sample KS3 tests it would have been helpful to know how successfully such questions discriminated the four levels of achievement for the tests, but no such information is provided in the sample materials. The first question is one of three - for a total of four marks - aimed at the lowest level of achievement for the tests (AF2: information retrieval): From the first paragraph, write down the month and precise time at which the experiment is taking place. - as plainly stated in the onset of the first and third sentences:- It was on a dreary night of November… It was already one in the morning… How productive a question was this: did significant numbers at the lowest level in the pre-tests really manage to misread the time as the 'one pm', as suggested by the mark scheme (not that 'already one in the morning' indicates the 'precise time')? Unless significant numbers at particular levels fail questions we are not testing anything. We leap levels for the next question (AF4), if still for but one mark:- In the first paragraph, how does the way the final sentence is written build up tension? - with a number of stylistic features to chose from, for the markers that is, not for the pupils, who have to predict as best they may what kinds of things are looked for: AF4: identify and comment on the structure and organisation of texts, including grammatical and presentational features at text level But there is a mismatch here in any case, since, while the tension is built up, at text level, in the narrative context of the whole paragraph - with the indications of stress in the preceding sentences ('dreary', 'labours', 'anxiety', 'agony', 'lifeless thing'…) - all the features noted in the mark scheme are internal to this final sentence, and so at sentence level. But we are now confronted by one of the fundamental issues in this kind of testing: the assessment focuses used by the setters and markers are not made available to the pupils. If such questions are to be asked - if - then it was some virtue at least of multiple-choice testing to make the marking criteria more explicit in the sets of possible answers in the question formats, and so to concentrate setters' minds on what was actually being asked. Question 3, similarly, also on AF4, would have been better set in a multiple-choice form, but at least it looks at inter-sentence features, if only for one mark in picking contrasts and one for a comment on their effectiveness. And that is a further problem with a handful of one-mark questions; they either question trivial points or trivialise more significant points. The fourth question on this first passage does ask for some discussion - twenty lines allowed in the answer booklet, for five marks - on the writer's 'use of language', in which pupils are asked to 'comment on how the writer': · sets
the scene This is under AF5, which includes 'grammatical and literary features at word and sentence level', though this is not made explicit in the question nor covered in the mark scheme by anything more than 'Some relevant words and phrases, suggested by the prompts [in the question], are clearly identified'. It is a similar pattern for the four questions on the second passage, with a minor piece of commentary set similarly as the fourth question (8), but under AF6 - as for the final question (14) on the third passage (above), except that no writing is required at all on the third passage. 'Writing' is set as a separate paper.
This comprises one piece of writing, contextualised in terms of audience and purpose in a given scenario: As a year 9 pupil, write a speech to give at this discussion evening, analysing what you think are the advantages and disadvantages of teenagers having part-time jobs. A planning page is provided: · words
and phrase to begin and end your speech But how long a speech? The only guidance given is the timing (45 minutes, including 15 minutes' planning time) and the total marks (30). And a written speech - as an appropriate model for either speaking or writing in Year 9? We might ask pupils to write notes for a speech - assessing the ensuing speech, not the notes - or to write a piece (in so many words) for, say, a newsletter, but this bogus task is neither one nor the other, and no supported personal opinion is sought. The guidance on the three strands of the assessment given to the markers: A Composition
and effect (AF1 and AF2) - 8 marks - again withheld from the pupils - gives the greatest weight to those surface features in writing of least significance in a script for speaking, supposing this were 'making it real'. The Shakespeare paper But Writing is also the objective of the first task (Section A) in the Shakespeare paper - for over half the marks (20): … short tasks are set which are designed to elicit succinct responses… [relating to] themes and ideas arising from pupils' study of the set play… linked to one of the designated writing purpose triplets in the English Order… : Henry V: inform,
explain, describe; - not that the pupils are to know this - and the Twelfth Night Writing task is similarly misconceived: Write what you are going to say in assembly. (Twelfth Night) For this half the marks are for 'Sentence structure, punctuation and text organisation' (6 marks) and 'Spelling' (4 marks), again, largely those surface features of writing of least significance in a script for talking. There are further levels of unreality in the Twelfth Night task: Imagine you are a modern-day Malvolio, in charge of preventing any sort of fun and enjoyment at your school. 'In charge of…' - is that even remotely likely? There are real issues for 14-year olds to address about allowing 'chips in the canteen', and about the roles of 'singing/music' and 'games/sports' in the curriculum, but the other options put up for banning as 'fun or enjoyment' - 'jokes' and 'smiling' - are simply as silly as the task set, for which the play studied is the barest excuse. It is 'smiling' that gives the game away in this bogus scenario. Malvolio is certainly fooled into attempts at smiling himself, but it is not actually something he is shown to have condemned in others; this is merely a specious nod to the play that manages to get the reference wrong into the bargain! Are the pupils supposed to refer to Malvolio in their remarks? No reference to the play is required; the only tangential feature is that humour is required in the role for the higher marks under 'Composition and effect' (10 marks), but how are they to know? The scenario for the Macbeth Writing task is also bogus: supposedly a request to 'The Publicity Department' for 'a publicity booklet for teachers' for 'a video of Macbeth in modern dress'? In the absence of any other information offered about the video the pupils are asked to outline: · what's
on the video;
But what writing? A leaflet might be a single sheet, possibly folded; a 'booklet' presupposes several pages, a length way beyond the scope of a 30 minutes' task. No target length is indicated, though there would be a tight word limit in a real publicity brief, with constraints on the writing and format imposed by graphics (for a video publicity leaflet!), layout and house style. In short, a more realistic task for group work using ICT - after watching such a video - but quite unrealistic as a short hand-written piece in an individual test task when none of this applies. All the pupils are asked to do is to 'Write the text', but that begs all the questions. The mark scheme does make some concessions to the use of an informal style - though, again, how are the pupils to know? - but it is prose paragraphs that seem to be required. The context provided for the Henry V Writing scenario also undermines the task set: 'The library is producing a collection of writing called "People We Admire"… Write your contribution…' No length noted, but in half an hour the pupils are expected to: · choose
a famous person or someone you know; - all without the access to library resources that would be the focus of such an invitation. And the tangential link to the play only confuses the issue: When watching Henry V, the audience might admire the king as a strong leader and a hero. What about heroes who oppose strong leaders (the 'William Tell' factor), for instance? The play is irrelevant to the task set. And that is the nagging question about these 'Writing' tasks set in the 'Shakespeare' paper: what have they got to do with the study of the plays? That is down to the 'Reading' tasks - with more time advised (45 minutes, allowing 15 minutes for reading the extracts - if not suggested?) - but for fewer marks (18). The sample tasks are set on: · the
language of the text (Henry V) - in conventional enough extract questions, though with some unreality again in the 'performance' scenario set for the Macbeth scenes: Imagine you are going to direct these extracts for a class performance. Explain how the actor playing Macbeth should show his reactions, and give your reasons. What is a 'class performance'? 'Direct' and 'actor' suggest some kind of presentation for an audience rather than the more productive workshop exploration of the scenes more likely in a classroom context - proving something of a distraction in the mark scheme in any case when the evident objective is close textual discussion. Making it real? So, the dominant impressions of these sample KS3 English tests: · pseudo
reality in contrived contextualisation of tasks in Reading and Writing Go
to LATE home page
|