Editorial
Rachel Forsyth

The Concept of Plagiarism
Bill Johnston

Plagiarism Detection Software - a new JISC service
Rachel Forsyth

Can students assess students effectively? Some insights into peer-assessment
A. Mark Langan and and C. Philip Wheater

Exploring the potential of Multiple-Choice Questions in Assessment
Edwina Higgins and Laura Tatham

Developing a new assessment strategy
Gaynor Lea-Greenwood

Assessing the Un-assessable
Tim Dunbar

How to assess disabled students without breaking the law
Mike Wray

Returning Feedback to Students via Email Using Electronic Feedback 9
Phil Denton

Tools for Computer-Aided Assessment
Alan Fielding and Enid Bingham

Faculty Learning and Teaching Reports

Learning and Teaching News from the Library

| View this article as a .pdf file |

Edwina Higgins and Laura Tatham
School of Law

Exploring the potential of Multiple-Choice Questions in Assessment

Abstract: This article explores the potential for using MCQs for assessment in a subject discipline in which MCQ testing is not universally accepted as valid. Some of the advantages and misconceptions associated with MCQ tests are explored, and in particular we look at ways of testing more than knowledge-acquisition. Two examples are drawn on: a paper-based summative assessment on the Post-graduate Diploma in Law and web-based formative assessment exercises in WebCT on a unit on the first year of the undergraduate law degree. We conclude that the potential for web-based formative assessment makes it worth investigating the use to which MCQs can be put in testing intended learning outcomes, and make suggestions for further reading.

Why MCQs?

We have to start by admitting that our interest in multiple-choice questions was not sparked by an intellectual debate over the best way to measure our intended learning outcomes but by an addiction to the TV Show Who Wants to be a Millionaire? (we do). That said, there were two more ‘educational’ reasons why we started investigating the potential for MCQs, and we were coming from slightly different angles. First, Laura ‘inherited’ subject leadership of the English Legal System unit on the Postgraduate Diploma in Law in 1999 (this is the law ‘conversion’ course which is the first step for graduates of other disciplines who want to qualify as legal professionals). Students are required to show competence in this unit before being allowed to proceed with the rest of the programme, so the marks for this unit must be produced almost instantaneously and traditionally this has been achieved by a knowledge-based multiple-choice assessment, which also addresses the problem that these students are new to law and have not had time to develop their legal writing skills. Secondly, Edwina has a particular interest in developing methods of web-based learning using WebCT, and we are both involved in a project to provide greater online support to the students studying the PGDL programme by distance learning. In particular, Edwina has been working on (some might say obsessing about) developing more effective means for students to test their understanding and receive feedback online. MCQs are one obvious mechanism for this, but we did not want to fall into the trap of using MCQs merely because the technology facilitates it.

Therefore from two different perspectives both of us are interested in multiple choice questioning and the questions we wanted to address were:

1. In summative assessment, could MCQs, adopted for ‘practical’ reasons because of a short marking turnaround time, really test whether students have achieved the application elements of the intended learning outcomes of a unit?
2. In formative assessment, to what extent do MCQs offer scope for effective self-testing exercises and feedback in a web-based format?

So, we needed to know more about MCQs. This may sound odd to colleagues in other disciplines who have been using MCQs for years. MCQs are in use for a variety of tests ranging from driving tests to standard attainment tests for schoolchildren (not to mention quiz programmes). Within legal education, the validity of multiple-choice questioning as an assessment is upheld by the America Bar Association, and all UK providers of the Bar Vocational Course (bar one) use MCQ tests to assess all non-skills subjects. Within undergraduate legal education however, poor old MCQs still have something of a bad reputation. Yes, they are easy to mark, but you can’t test more than surface learning of facts with multiple-choice questions, we were told; an averagely lucky monkey could expect to get reasonable marks in an MCQ test.

We wanted to explore whether this reputation was deserved. Could we use MCQs to assess effectively the skills and understanding which we wanted to test? If we could, then this would address Laura’s concerns about the effectiveness of her summative assessment and open up the potential for Edwina’s online tests to provide an effective means for formative assessment. We discovered that some legal educators¹ have argued that MCQs can be used to test the same sort of skills as ‘traditional’ problem-based learning in law. What we also discovered, after speaking on this topic at a law teachers conference, was that colleagues in other institutions were keen to explore the potential for using MCQs if these same concerns could be addressed. It seems the efficiency of MCQ tests makes them an attractive proposition, whether because of increasing use of multimedia learning tools or because of the practical pressures of ever-increasing student numbers placing greater assessment burdens on staff – as long as this isn’t a move towards ‘dumbing-down’ or encouraging rote learning.

Investigating the myths

Multiple-choice questions are credited with a number of advantages. They are objective, so variations in marking due to subjective factors are eliminated (although the questions themselves still have to be scrutinised to ensure that they are not biased) and this also makes them easy to mark² (and they do not necessarily require an experienced tutor to mark them). This is likely to be a significant advantage for over-burdened staff. They are efficient because questions take less time to complete, and therefore it is possible to test a greater range of the syllabus through setting compulsory questions (there seems to be an inherent assumption that MCQ tests will feature compulsory rather than optional questions)³. Nevertheless, there are a number of potential disadvantages associated with MCQs, and these are considered here.

1 Does adopting MCQ assessment mean ‘dumbing down’?

This seems to be the major concern raised about MCQs. They have a reputation as being “easy” and there is certainly a danger that they may be – think about those £100 questions on Who Wants to Be a Millionaire? – but any form of assessment can be designed for a particular level of difficulty. With MCQs, the difficulty level depends upon the nature of the questions asked and also how the multiple-choice questions fit into the overall scheme of assessment. Careful attention needs to be paid, as with any assessment mechanism, to the ability of the questions to test the intended learning outcomes set for the students. In a subject discipline where application and writing skills are required, replacing the whole of an assessment strategy with multiple-choice tests comprising simple knowledge-based questions would not allow an adequate assessment of the required achievement, which would rightly be regarded as dumbing-down – but the same questions might be extremely useful for formative assessment throughout a unit, or as part of the summative assessment. Have you ever despaired, like us, at the desire of students to “question spot” by ‘learning’ only a limited portion of the syllabus? A traditional end-of-year examination where students answer a small number of questions out of a range covering the whole syllabus (a very popular model in legal teaching, and one often regarded as the “proper” way to assess student achievement in law) arguably encourages them to do so. Having a compulsory MCQ section allows efficient testing across a greater range of the syllabus – if that is what you want to do – which is probably a more acceptable alternative to setting a large number of compulsory essays or problems.

Additionally, multiple-choice questions need not be limited merely to testing knowledge; tests can include more challenging comprehension/application-based questions, and these might well allow more of the intended learning outcomes for a unit to be assessed efficiently and thus be valid for summative assessment. In other words, MCQs are versatile, and it is only if they are inappropriately used or poorly designed that there is a risk of ‘dumbing-down’. Some issues of question design are considered later⁴.

2 Are MCQ scores unrealistically high?

Also contributing to the reputation of MCQs as “easy” is that the scores tend to be higher than those attained in non-objective assessments in our experience. This exposes the discrepancy between assessments which use the full marking range (0 – 100 per cent) and more traditionally-marked assessments (in undergraduate law at least) where there may be a ‘glass ceiling’ around the 80 per cent mark. This may be problematic where an MCQ assessment is used within a diet of conventional assessments. Questions must be carefully designed to ensure the same level of achievement is required for a pass in an MCQ test as for other comparable assessments (although it is really the subjective ‘glass ceiling’ effect which is at fault, rather than the MCQs which are, after all, objective).

High scores pose less of a problem where the MCQs are used as part of formative assessment, but students may have optimistic expectations of their abilities if they have scored highly in MCQs throughout the unit but are then faced with a final formal assessment asking them to display different skills. The purposes of any formative assessment (and the summative assessment, if different) need to be made explicit to students.

3 Can students guess their way to success in an MCQ test?

A second scoring issue is that a student has a theoretical chance of “guessing” the correct answer in an MCQ – the ‘lucky monkey’ problem. Arguably this is no worse than a student who adopts the write-all-you-know approach to a question for which s/he can generally expect to pick up marks for the correct points the marker has laboriously identified within the largely irrelevant answer⁵. Nevertheless this does present a problem to which there is no definitive solution, although there are various possibilities which can be adopted⁶, including:

• use of negative marking to discourage students from equating multiple choice with multiple guess;
• adopting mathematical strategies to “normalise” marks achieved;
• ensuring there are sufficient options (answer choices) for each question and/or raising the overall pass mark for the test to reduce the likelihood of a student passing through chance.

Even if these solutions don’t sound satisfactory to make MCQs a valid option for complete summative assessment, the problems are less significant for formative assessment (although, as already mentioned, care does need to be taken about false student expectations where the strategies for formative assessment differ from the summative assessment). However, the advantage of objectivity in MCQ summative assessment should be stressed. Traditional forms of assessment, for example essays, raise considerable problems in terms of subjective marking, even where assessment criteria and moderation strategies are adopted, and the phrase “it’s an art not a science” defensively employed. One of the lessons we learned was that an assessment method shouldn’t be rejected just because it is new or different in a particular subject discipline, any more than another is automatically valid simply because it is the one that has been used for years. In many cases carefully designed MCQs that are challenging and require satisfaction of (all or part of) the intended learning outcomes may be sufficient to reduce the effect of the scoring problems outlined here.

4 Does using MCQ assessment encourage rote or surface learning?

In our experience some students will rote-learn whatever method of assessment is adopted. Legal ‘problem’ questions allow examiners to distinguish between surface learners and those who have shown understanding through a higher level of application and analysis, and hence avoid rewarding (and by implication encouraging) surface learning. We wanted to set questions which would allow students with greater application and analytical skills to shine. Appropriate MCQs can be used in the same way to distinguish the surface learner from the deeper learner, where this is the purpose of the assessment, by setting questions which require comprehension and application skills, although testing requires a sufficient number of questions to eliminate those lucky monkeys. We have to remember however that any form of valid assessment is likely to require the student to display some form of knowledge acquired by rote learning. Knowledge-based questions are also a useful formative assessment tool to check whether students have grasped the basics. We found that giving students a sample of the type of questions they could expect was the best way of encouraging them to adopt appropriate learning methods.

5 Can MCQs test oral and written skills?

This may well be a drawback for colleagues considering incorporating multiple-choice questioning in their teaching. MCQs can test much more than knowledge but they cannot test oral or written skills⁷. They cannot test that students can form appropriate arguments and discuss the subtleties of a topic. Where the intended learning outcomes of a unit require oral or written skills a multiple-choice test cannot form the entire assessment. However, MCQs offer ample scope to test much of the knowledge and skills required by the legal curriculum and can therefore be extremely useful for formative assessment purposes and, where appropriate, as part of summative assessment. We decided to concentrate on what we could test rather than on what we couldn’t!

Features of good question design

Many of the potential problems with MCQs identified above can be ‘designed out’ with well-written questions. Poorly designed questions may:

• Give away clues to the answer
• Fail to test the skills required by the intended learning outcomes
• Contain “implausible distracters” (obviously wrong answers) which can be eliminated by students with only limited knowledge
• Encourage rote learning
• Confuse or frustrate students with sound understanding

All of the above problems could result in scores which do not reflect student ability.

There are numerous guides to writing effective multiple-choice questions⁸, and we don’t have scope to go into more than a few key points here. A crucial factor will be the ‘distracters’ which are written, and Who Wants to Be a Millionaire? again provides us with an illustration. Recently the following question came up at the £100 stage: Which US state was the third state to accede to the Union in 1787? To us at least (!) this is a comparatively difficult question which certainly had the contestant gulping, but the level of difficulty became laughable when the options were revealed, which were (a) New Cardy (b) New Woolly (c) New Jumper and what you have probably already realised is the correct answer (d) New Jersey. We use this to emphasise that the art of providing likely wrong answers (or plausible distracters to give them their correct term) is a key factor in writing good MCQs⁹. The same question would have been harder if the options had been, say, New Hampshire, New Jersey, New Mexico or New York because all four are actually states. Here the difficulty lies not with the question stem itself but with the plausibility of the distracters.

It should be emphasised that tutor perception of the plausibility of a distracter may differ widely from that of students taking the test. We once set a question about the burden of evidential proof in a criminal case which we intended to be a fairly simple knowledge-based ‘opener’ to a test. The scenario involved a man who had been arrested for an offence involving a musical monkey, and we were dismayed at the number of students who felt that the burden of proof in this case would be on the monkey… (What this says about our teaching is quite possibly more depressing.) One of the best ways of writing plausible distracters is to utilise common student mistakes. This forms a “trap” for weaker students but allows students with better understanding to demonstrate it. The use of distracters based on common misconceptions in web-based formative assessment situations is explored below.

A further trap which needs to be avoided is the temptation to make a factual MCQ ‘harder’ by simply testing more obscure knowledge (think about the top-money questions on Millionaire). Factual questions need to be related to the intended learning outcomes for the course. Best practice suggests that designing harder questions should involve moving through Bloom’s Taxonomy from knowledge-based questions to questions which demand comprehension or application¹⁰. Application skills are a particularly important aspect of legal study, so this transition was something we were keen to develop in our examination for the PGDL English Legal System unit.

Developing the Multiple Choice Examination for English Legal System

This unit had adopted an MCQ test for its assessment because of the quick marking-time. Students must show competence in this unit before being permitted to continue with the remainder of the programme, so it is imperative that the results are released quickly. With over 150 students on three modes this is a heavy marking burden on the unit team. Student on the Distance Learning mode, for example, take the test on the Friday afternoon of the first of only four attendance weeks throughout the year and the staff team release their results within an hour of the test’s end so students know whether they can continue with the remainder of the programme before leaving Manchester.

Given the ‘myths’ identified above, we needed to investigate the validity of the MCQ assessment. We were not too concerned about the scoring issues because in this assessment we are simply looking for competence; we do not need marks to relate to a percentage or classification. We readily admit that this allows us to side-step one of the key problems with MCQs identifed by many authors. As far as the limitations for testing oral and written skills are concerned, again this was not a worry as these were never part of our intended learning outcomes. These skills are acquired and assessed later on in the programme.

The issue which did concern us was how to ensure that the unit’s intended learning outcomes – one of which is legal application skills – were assessed. How could we set a test which would demand more than just knowledge repetition? We devised a strategy which involved the students having to work through a set of materials additional to the MCQ test paper itself during the exam. The additional materials comprise a fictitious Act of Parliament and a series of scenarios which have led to prosecutions under the Act, including extracts from the (also fictitious) legal judgments for each ‘case’. The students then answer multiple-choice questions which test their knowledge of the legal system, their comprehension skills in relation to the materials given and their ability to apply their knowledge to the scenarios and draw conclusions about the connections between the cases. This method of assessment allows us to retain the fast turnaround time we need, while testing the students’ fulfillment of the ‘higher level’ intended learning outcomes of the unit.

Developing formative assessment exercises using WebCT

Producing an exercise for English Legal System which allowed us to test comprehension and application rather than just knowledge proved that MCQs could provide a useful tool for assessing the skills law students must acquire. This was an interesting prospect because it opened up the potential for creating formative exercises for students to access online (of course formative exercises don’t have to be online – it could be an oral run-through of answers following a test or written feedback on the questions, where students can identify for themselves where they went wrong – but the great advantage is that tutors don’t have to mark web-based questions). In contrast to essay or problem assessments where feedback is laboriously dependent on individual answers, the closed answer-range for MCQs means that tutors know all the possible errors that can be made in advance of the test being completed, so MCQs offer an excellent opportunity for the tutor to provide generic yet focussed feedback. In particular, the format of MCQs is ideally suited for delivery via a web-based learning environment, which means that students can complete assessments and obtain marks and feedback immediately independent of tutor-contact.

Our first year undergraduates study a ‘frontloaded’ unit called Introduction to the Study of Law for the first four weeks of the programme, and this is supported by a WebCT site developed by Edwina in the distant days of a university online fellowship. It contains revision ‘tutorials’ to consolidate the work done in the classroom, provide further tips for completing their portfolio tasks and help students revise for the end-of-unit exam. The students complete part of their assessment by submitting a WebCT quiz, which is a knowledge-based test about the legal system – the point of the exercise is not so much that they know the stuff but that they have been able to find it within certain web-based materials, to familiarise them with the amount of legal information available on the internet (which is one of the intended learning outcomes). What we particularly wanted to do, however, was to develop formative exercises using the self-test facility in WebCT. This allows students to get instant feedback, and – unlike the quiz facility – WebCT does not record a ‘score’ for the student, so they can complete the exercise without feeling self-conscious. Students can have as many goes as they like at the self-test, either immediately or after further revision.

A particular advantage of using the self-test facility is that it helps to make the revision tutorials on the WebCT site more interactive; instead of just reading material which (except for the pretty pictures) could basically have been given in a handout, the students can check their understanding at regular points throughout the revision tutorials and have the opportunity to engage with the material rather than just staring at the screen. There is more to using the site than just pressing the ‘print’ button! Sometimes, the self-test exercise may only be one question, as for example with the self-test reproduced in figure 1, which accompanies a page in which definitions of the legal term “ratio” were explored with a practical example. The students have to be able to distinguish ratio from the related concepts of ‘obiter’ and ‘decision’. None of the options given is a ‘standard’ definition of ratio, so this tests students’ comprehension of the material.

Figure 1

Here, what we were trying to do was to maximise the effectiveness of the feedback given so that students would be able to improve their understanding to reach the right answer (although not all our self-test questions have the quantity of feedback in the illustration). Effective feedback is the key to formative assessment, so it is vital to devote time to writing feedback from which students can learn and which encourages them to think and evaluate their level of understanding: it does not simply tell students where they went wrong but why. Well-written questions with plausible distracters based on common student mistakes are particularly useful in communicating to students the gaps in their knowledge or understanding (all the distracters in Figure 1 are based on common errors about the concept of ratio). The more plausible a distracter is, the easier it is to write feedback for it; the better the feedback, the more the students learn from it.

Some useful pointers for feedback include:

• Giving feedback which is as detailed as possible – “No” or “Wrong” does nothing to help the student identify the reason for their mistake, although with some simple factual questions, and with less plausible distracters, it can be difficult to do otherwise;
• Writing feedback at the same time as the questions rather than trying to do it later helps to focus attention on common student mistakes and how they can be corrected;
• If the question is web-based, directing students to the resources required for them to correct their understanding and try again, rather than simply giving them the correct answer;
• Where appropriate acknowledging, with encouraging feedback, those students who have answered questions well, or who have got something ‘nearly right’ by choosing a particularly plausible distracter.

Conclusions

This article probably seems of little value to those in subject areas where MCQ tests are a standard part of the curriculum – we know we are beginners – but the intention is to evaluate their potential in areas where they have traditionally been rejected because of the myths we identified earlier. MCQs are not, ultimately, a mechanism for reducing the marking workload (if only!); there are skills they simply cannot test and care needs to be taken with the scoring scheme. So why bother to consider MCQs? Exploring the use of objective questions really opens up the potential for web-based formative assessment so students can get instant feedback and ‘individual’ (albeit automated) feedback on their understanding. Having the staff resources to provide this individual support face-to-face is an increasingly rare luxury as student numbers grow. Of course, feedback from a PC screen isn’t the same as from a nice human tutor, but our student evaluation forms show most of the students find it helpful. Some even prefer it, because it has the advantage of flexibility; students can interact at the times that suit them. Online exercises are a valuable way to transform web resources from handout graveyards to something more engaging.

Our enthusiam for both our MCQ experiments has led to some cross-fertilisation. We now set the LL.B. Introduction to the Study of Law students an MCQ application exam in a similar format to the PgDL English Legal System one (though it is a bit easier) and the ELS students now get access to a WebCT site with formative exercises similar to the ISL ones. The two original ‘angles’ we set out to investigate have come together in provision for both sets of students of an exercise on WebCT in which they can work through a previous year’s MCQ test and additional material, and obtain feedback on their progress via a WebCT self-test. Our next project is to consider how a standard ‘problem’ question in law could be broken down into MCQs to provide formative web-based exercises in other legal subjects, to give greater support to our distance learning students. Inevitably, the main stumbling-block is finding the time.

If you are thinking of developing MCQ exercises, the diagram in Figure 2 and the resources list below may be of help.

Figure 2

Issues to consider before designing an MCQ test

Guides to designing MCQs:

Carneson et al, Designing and Managing Multiple Choice Questions. http://www.le.ac.uk/castle/resources/
Explores the relationship between different types of question and Bloom’s taxonomy.

Kehoe, Writing Multiple-Choice Test Items. http://www.ed.gov/databases/ERIC_Digests/ed398236.html
A readable guide to question design, containing useful tips and analysis.

McKenna C and Bull J (1999), Designing effective objective questions: an introductory workshop http://caacentre.lboro.ac.uk/dldocs/otghdout.pdf
The CAA’s useful guide to question design, with examples of tested questions demonstrating alternative formats.

Other sources

Alldridge. P (1997), Multiple Choice Examining in Law 31 Law Teacher 167
Describes the advantages and disadvantages of introducing multiple choice examining in a law module.

Bush M, Alternative marking schemes for Online Multiple Choice Tests. http://caacentre.lboro.ac.uk/dldocs/BUSHMARK.pdf
Considers scoring options for MCQs, including use of negative marking and normalisation.

Bobb-Semple C et al, Inns of Court School of Law: Test Yourself in Evidence, Civil Procedure, Criminal Procedure, Sentencing, 4th ed. London: Blackstone Press, 1997.
The BVC text containing sample questions and an analysis of misconceptions in multiple choice examining.

Log on to student motivation. The Times Higher Education Supplement May 3 2002
Describes the use of formative web-based MCQs to complement face to face teaching

Paxton M, (2000) A Linguistic Perspective on Multiple Choice Questioning 25 Assessment & Evaluation in Higher Education 2
A critique of the emphasis on multiple-choice testing as a response to increasing student numbers in South Africa

Quanjel-Schreurs L et al (2002), A Digital Programme for Advocacy 36 Law Teacher 15

Notes

1 See for example the testing of inchoate offences described by Alldridge P., Multiple Choice Examination in Law (1997) 21 Law Teacher 167

2 Jolliffe, Ritter and Stevens, The On-line Learning Handbook - Developing and using Web-Based Learning. London: Kogan Page, 2001.

3 Ibid.; see also Simes et al, Inns of Court School of Law Test yourself in Evidence, Civil Procedure, Criminal Procedure, Sentencing, 7th Edition. London: Blackstone Press, 1997.

4 For examples of ‘poor’ and ‘improved’ questions, see particularly McKenna C and Bull J (1999), Designing effective objective questions: an introductory workshop [online] http://caacentre.lboro.ac.uk/dldocs/otghdout.pdf

5 Alldridge, n1 supra, points out that a further advantage of MCQs is that they eliminate the potential for students to adopt what he calls the ‘Dresden’ approach.

6 See for example Bush, M., Alternative marking schemes for online multiple choice tests [online] http://caacentre.lboro.ac.uk/dldocs/BUSHMARK.pdf

7 But see Quanjel-Schreurs, L. et al, A Digital Programme for Advocacy (2002) 36 Law Teacher 15, on the possibilities of learning by modelling in skills-acquisition.

8 See the resources list at the end of this article for suggestions.

9 Particularly recommended for further advice on writing distracters is McKenna C and Bull J, (1999),Designing effective objective questions: an introductory workshop [online] http://caacentre.lboro.ac.uk/dldocs/otghdout.pdf

10 See Carneson et al, Designing and Managing Multiple Choice Questions [online]. http://www.le.ac.uk/castle/resources/

Edwina Higgins
School of Law
0161 247 2431
e.higgins@mmu.ac.uk

Laura Tatham
School of Law
0161 247 2430
l.tatham@mmu.ac.uk

February 2003
ISSN 1477-1241