Measuring student workloads is part of a wider debate that the University will have to engage with in the near future, namely the tension between the British “learning outcomes” philosophy and the “time served” philosophy which underpins the Bologna process. The conflict between our “learning outcomes” philosophy and measuring workload is one of the reasons that I did not, and cannot, identify simple assessment workload guidelines.
In terms of the National Student Survey (NSS) scores there is little doubt that the Open University excels. It is, then, perhaps surprising that they do not have well established workload protocols
“this report has highlighted the wide diversity both within and across faculties in how it should be assessed and addressed”
( Course Managers and Student Workload: Report and Executive Summary).
This suggests that there is no simple answer and a suite of recommendations may be needed to replace hard guidelines.
In the few studies that have attempted to measure actual student effort it is clear that there is considerable variation and student effort may not reflect the marks allocated to an assessment. Inevitably differences between students (knowledge, skills, time) influence their perception of the difficulty of assessments. It is unsafe to assume that students spend longer on assessments that they consider to be the hardest. It is also unsafe to assume that students who spend longer studying feel more overworked.
There is clear evidence that there are subject level differences in assessment practices some of which are passed down as ‘folklore’ whilst others arise from professional body requirements.
There are two assessment workload metrics in common use: estimated assessment hours and word limits. Word limits are very difficult to apply to assessments which do not adhere to the essay or report format. When word limits are translated to student hours there is little variation within MMU, within Britain or even internationally. Nine recommendations were approved by the Faculty Academic Development Committee (FADC) in Science and Engineering.
As part of my work as an SLTF in Science and Engineering I was asked to undertake a review of student assessment workloads with the aim of providing guidelines for departments and programme leaders. As a starting point the literature was reviewed and programme leaders invited to attend a workshop (12th Dec 2007) led by Rachel Forsyth (CeLT). Eight staff from three departments attended and discussed some of the key issues. Staff associated with the project also attended university assessment workshops where similar topics were discussed. I also reviewed practices within MMU and in other universities within the UK and elsewhere in the world. At the end of this process a series of recommendations were made and subsequently approved by FADC.
Prior to the workshop a discussion document was prepared and circulated. This is described later. During the workshop, staff from each department described how they currently, or historically, attempted to ensure a consistent assessment workload. It became clear that there were large differences that would be difficult to reconcile, largely because of subject-specific factors. It was also clear that it would be very difficult to design a metric that measured student workload across the diverse range of assessments used in the faculty.
The Divisions of Biology and Health Sciences within the School of Biology, Chemistry and Health Sciences, have a well defined student assessment workload model that is applied consistently across all programmes. In another department the allocation of effort to student assessments was in transition, mainly in response to inconsistencies between external examiners that arose, partly, from the wide range of disciplines covered by their courses. The third department does not use a word count model because this is not appropriate for their assessments. As a guideline, they aim for three assessments per 20 credit unit. They have experienced some comments from students about differences in workload, particularly when comparing the ‘creative’ versus ‘technical’ units.
The general feeling of those present was that they did not think that a faculty-wide set of student assessment workload criteria could be established, mainly because of the large differences in assessment styles between subjects and the ensuing difficulty in measuring workload. However, it was agreed that programme teams need to consider student workload and demonstrate, as far as possible, that workloads are equitable across units within a programme.
As a starting point I examined the rationale for assessment workload guidelines and identified ten key areas.
There is a general acceptance (national and international) that 1 Credit Point equates to 10 hours of student effort. (except for the European Credit Transfer System (ECTS) where comparisons become difficult). Therefore, a 20 credit unit = 200 hours student effort. But, should the assessment load be measured in hours or words? Word count is a very simple metric but it is not easily transferable across different assessments and word count does not equal difficulty. It is possible to imagine a 1000 word assignment that is more intellectually demanding than a 2000 word essay. Student working hours are more general, i.e. can be applied to a wider range of assessment activities, but how long should a particular task take and do our estimates match the actual time taken by students? Crook and Parkes (2004) used an excel-based study diary to measure the actual time taken and they found that
“Interestingly, there was no significant relationship between the time students spent preparing for assessed coursework and how much that coursework contributed to the module’s overall marks (P=0.370). In fact, students were in some cases observed to spend more time preparing for assessments that were cumulatively worth 20% or 30% than they were preparing for assessments within modules that were 100% coursework-assessed. As expected, there was a normal variation in the overall Part two mark achieved by students. However, if preparation time per percent mark achieved is calculated then there is considerable variation between students (ranging from 1.6 to 6.7 hours). This is again not surprising given the nature of individuals, but it does illustrate one of the difficulties of trying to allocate notional study hours to large and diverse (in terms of module choices) groups of students.”
Kember and Leung (2006) carried out a large study (n = 3320) in a Hong Kong university. The following quotation provides an overview of their findings and backs up the findings of Crook and Parke.
“The inferences were that perceptions of workload were very weakly related to the number of hours spent in class or studying outside class. Two of the case study students each worked for a total of 64 hours in the week examined. Despite these very high study times, their workload perception ratings were only a little above the mean. Both students were sufficiently interested in their course to devote much of their waking hours to study, without finding this too stressful. One of the two commented, ‘the workload of this course is not very great but is time demanding’ This student devoted 32 hours during the week to work on a group project, and the other also had project work. By contrast, the student who devoted the least amount of time to study (9 hours of classes and 13 hours of independent study) had the highest perceived workload rating. He lacked interest in his course. The teaching in this was predominantly didactic, with little interaction between teachers and students. Assessment tended to stress the ability of students to recall model answers in tests and examinations. Rather than being a measure of time commitment, perceived workload was shown to be a complex construct which could be influenced by a wide range of aspects of teaching and learning. The influences appeared to be interrelated, and so acted in concert as a holistic environment, rather than there being simple correlations with individual variables.” (My bolding)
Workload models were examined at four levels: within the faculty; within MMU; in the UK and internationally. Information was obtained by web searches and cannot be considered to be exhaustive. However, all institutions, for which an articulated policy was identified, were included. It is interesting that there are relatively few. It is also recognised that some of these policies may no longer be operating, or have been subject to amendments.
BCHS (Ex-Biological Sciences)
A 20 credit unit has 200 h of associated student effort. This is most appropriately regarded as:
The appropriate unit of currency for assessments is not marks per assessment, but the student effort associated with each assessment. Table 1 shows the initial guidelines used by staff to calculate and justify the assessment model for each unit. If staff intend to use assessments other than those listed above they are asked to specify the number of associated hours of student effort. This assessment model was recognised as good practice during a recent MMU institutional audit
“The academic level required of students was explicitly linked to a clearly articulated teaching, learning and assessment strategy….”
Assessment | Student Effort | Total(h) |
---|---|---|
1 hour of examination (including unseen, time-constrained coursework) | 1h assessment plus 9h student preparation | 10 |
Seen essay or similar (1500 words) | 1h writing/typing plus 9h student preparation | 10 |
Practical report | Writing/drawing graphs 2h plus 3h student preparation | 5 |
Oral presentation (10 minutes) | 5h student preparation | 5 |
Poster (individual) | Assembly 4h (includes drawing figures etc), design 6h (including gathering and organizing information) | 10 |
Poster (group) | Assembly 4h (includes drawing figures etc), design 3h (including gathering and organizing information), group discussion 3h | 10 |
MCQ (1 hour) | 1h assessment plus 9h preparation | 10 |
Tutorial (1 hour) | Preparation and writing | 5 |
Environmental and Geographical Sciences
EGS used to work on the basis of word equivalents - a 20-credit unit would have 4 x 1500 word essays or equivalent. However, this became unworkable due to the wide variety of assignments set. They changed to a more flexible system based on student effort. Based on a 20-credit unit (200 hours) it assumed that there are about 50 hours contact, leaving 150 hours which should include everything else (e.g. all independent study, reading around the subject, reading/working in preparation for assignments, actually writing assignments etc). It was also decided that each 20-credit unit would have two assignments. There was opposition to the latter at the time because of the difficulties that it creates for giving feedback/early feedback. The two assignment rule has now been removed in the first year and is being phased out in other years for the same reasons.
Hollings
The Department of Food and Tourism use 1500 words plus a 2 hour exam for level 1 and 2. Level 3 use 2000 words plus a 3 hour exam.
Clothing, Design and Technology
The assessment tariff for 20 credits is 3 x 1500 word essays or a 3 hours examination (3 questions) or 24 (3 x 8) hours of practical (or pick and mix from these).
HLSS (from their ‘Yellow Book’)*
“At undergraduate level an element worth 25% of marks in a 20 credit unit shall be, or be equivalent in rigour and student learning effort to, assessed course work of approximately 1,500 words or a one-hour examination answer. Although not an exact science, as far as possible the terms used to describe elements of assessment should be consistent across the Faculty (see illustrative glossary below). ....... So, a programme team may assess a 20 credit unit entirely by assessed course work (for example, by four essays each of 1500 words, two papers each of 3000 words, one project-type extended piece of writing of 6000 words, a portfolio (requiring a variety of work), or entirely by examination. An element of assessment consisting of a one hour class test would normally be worth 25% of unit marks in a 20 credit unit.
Table 2 is a list of elements and their value is based on a standard 20 credit unit. It is for guidance, and is not prescriptive. However, good practice should encourage the Faculty to adhere to a common set of definitions.”
Book review 800 words | 10% of unit marks |
Short oral presentation (c. 5 minutes) | 10% |
Literature search 1000 words | 15% |
Seminar paper 1000 words | 15% |
Essay 1500 words | 25% |
Report 1500 words | 25% |
Portfolio 6000 words | 100% |
Project/short dissertation 6000 words | 100% |
Examination answer one hour | 25% |
Standard dissertation 12,000 words 40 credit unit, which may be assessed by a single percentage mark, or sub-divided into sections totalling 200 marks | |
Postgraduate paper 4-6000 words | 100% |
* Note that the faculty is reviewing its assessment practices as part of the Challenging Assessment Initiative |
The University of Southampton
Assessment Workload Revision 2 (25 April 2007) states that:
“AQSC believes that it is appropriate for schools to agree an appropriate assessment workload for students. It therefore does not wish to issue a directive about volume. However, each school is expected to publish what is expected of students and issue details in each handbook.”
London South Bank
Assessment Load: equivalence guidelines:
“Despite all the difficulties in trying to establish a ‘fair’ assessment regime it is still desirable that we do so thereby ensuring that student workloads are approximately equivalent for the same number of CAT points. The following table is based on the essay as a means of assessment. It assumes a norm of 200 – 275 words per credit point ie. 3000 – 4000 words per 15 CAT unit and offers suggested equivalences to 1000 essay words. It is derived from a review of assessment practice at other institutions. Where class participation is included within the unit assessment it must be defined and measurable, it should not exceed 10% of the marks for the unit.
Equivalent to 1000 essay words
* Examination or timed test 1 hour
* Essay in foreign language 300 words
* Group report 750 words per member
* Reflective journal or learning log 2000 – 2500 words
* Oral presentation 20 minutes
* Group presentation 10 minutes per member
* Clinical assessment 10 minutes”
University of Northumbria
Northumbria does not currently provide cross-University rules on the amount of assessment per module, accepting that this is a matter for academic judgement and that subject requirements will vary considerably. Historically, guidelines were provided when they moved to a modular system in 1993. They have a ‘one fifth of notional student workload’ guideline, which translates to 20 hours per 10 credits.
Extract from Northumbria’s recent guidelines
“It is not appropriate for these guidelines to define specific word limits for essays or lengths for examinations because of the varying needs of different subjects and varying requirements at different levels. It is likely that such guidelines will be produced at School level. The main point here is that staff are conscious of the assessment load they are imposing and set tasks appropriately.”
Bristol
An extract from their Assessment Guidelines and Degree Classification Annex A: Unit Structures and Methods of Assessment
Short Essay (1000-2500 words, depending on the nature of the task). Focused on a specific text or fairly limited topic, with an emphasis on specific knowledge and understanding. 20 hours research and writing
Long Essay (3000-5000 words). Dealing with a range of texts and/or multiple aspects of a complex topic, with an emphasis not only on knowledge and understanding but also on analysis, critical thought, development of original ideas etc. 40 hours research and writing
Unseen Exam. May be anything between 45 minutes and 3 hours in length; length to be determined by the nature of what is to be assessed, which also determines the nature of the questions set. The amount of time required for revision will be determined by the amount of material which will need to be covered, so that one might envisage 40 hours revision being required for a 20-credit unit but 80 hours for a 40-credit unit. 40-80 hours revision
Seminar Presentation. Depending on the length of the presentation and on the nature and complexity of the topic which students will be expected to cover, this may be considered analogous to either a short or a long essay. 20 or 40 hours preparation
Dissertation (8000-12000 words). Dealing in detail with a range of texts and/or multiple aspects of a complex topic, largely self-chosen and self-directed, with an emphasis on knowledge, understanding, critical thought, analysis, development of original ideas etc. It is impossible to prescribe a single notional workload figure for this exercise. Where a dissertation is used to assess a taught unit, a minimum of 80-100 hours seems appropriate. Where the dissertation effectively is the unit, placing greater emphasis on the student’s self-directed research and formulation of problems, workloads of 200, 300 or even 400 hours may be involved. What is important is that students should be clearly informed of the time they are expected to devote to the dissertation, and of the markers’ expectations in terms of content, scope, quality of argument, presentation and so forth, which are likely to be significantly higher than for other pieces of assessed work. Generally, dissertations are appropriate only for students at the highest level of study.
Leeds
(Philosophy)
Various assessment details are provided for their units but, overall, a 20 credit unit appears to be assessed by 4000 – 6000 word equivalents. For example, 50% 2000 word essay, 50% 2-hour exam or 2 x 2000-word essays, 50% each.
Open University
(Course Managers and Student Workload: Report and Executive Summary)
The Open University is renowned for its high scores on the NSS, even on those questions about assessment where most universities have problems. Consequently their Course Managers and Student Workload Report is very apposite, particularly since it was partly driven by retention concerns. It is a long, detailed and useful report and it is only possible to highlight several of their findings. It is a report that will benefit most readers and its conclusions informed my recommendations. The following bullet points are from their summary document.
Latrobe
(assessment document)
The norm is 250-300 essay words per CP, i.e. 3750-4500 words for a 15CP unit (equivalent to 5000-6000 words for a 20 credit unit). They provide a list of suggested equivalences to “1000 essay words”, including 2000-3000 words for an unstructured reflective journal.
University of Sydney
(Assessment of Graduate Coursework Units of Study - Guidelines)
(Note: 1 CP = 26 hours of student effort, so 20 MMU credits = 7.7 Sydney credits).
“To determine the size of assessment according to credit load for a unit of study, the suggested measure is 1,000 words of written product (or equivalent assessment) for each 1 credit. Equating one mode of assessment with another (e.g. seminar presentation, poster presentation, essay) will take into account issues of rigour and will be at the discretion of those developing the assessment but must be formally approved at the Assessment Programming meeting. Recommendations from the APM regarding assessment, will be made to the chief examiner.”
They also state that these are guidelines for the recommended maximum levels of assessment and that where:
“there is emphasis on other forms of student learning besides that related to assessment, the assessment load would appropriately be less than the above guidelines, whilst still meeting Master level standards of academic rigour.”
The listed examples include a 3-credit unit of study - report on on-line discussion group meetings plus reflective summary (equates to total 3,000 words) and a 4 credit unit of study - 1,000 word journal plus 3,000 word essay (equates to total 4,000 words).
In the UK, assessment is usually mapped to learning outcomes rather than directly to a workload model. For example, the Northern Ireland Credit Accumulation and Transfer System (NICATS) guideline 10 states:
“Credit is awarded for achieving learning outcomes. A learning outcome is a statement of learning achievement expressed in terms of what the student will know, understand or be able to do, on successful completion of the module. Each module has a coherent set of formally identified learning outcomes. In order to earn credit for the module, the learner must normally satisfy the assessment criteria for all [or the majority] of the designated learning outcomes for the module. “
Consequently, because it is usual to assess each learning outcome there can be a direct, although not necessarily strong, relationship between the number of learning outcomes and the amount of student assessment. However, as an advice note from the HEA Physical Sciences Centre notes:
“There are no rules on how many outcomes are appropriate per lecture course or credit point and any attempt to standardise would be completely artificial. Some modules may have many outcomes that are fairly easily achieved and assessed. Other, perhaps higher level, modules may have fewer, more complex outcomes which are more demanding to acquire and demonstrate.”
Other institutions, for example Queen’s University Belfast, provide clear guidelines and the Queen’s Open Learning Programme course proposal form states that the number of learning outcomes should be between two and four.
Irrespective of the differences in opinion about the appropriate number of learning outcomes it is reasonable to assume that a reduction in the number of learning outcomes will lead to a reduction in the number of assessments needed to test all of the learning outcomes. It is likely that this would lead to a reduction in the student, and staff, assessment workload. Consequently, when new modules are being designed programme teams should think carefully about the number of learning outcomes and how these should be assessed.
It is clear from the rationale for the need for assessment guidelines that we must ensure that we carefully consider the quantity and quality of our student assessments. This project attempted to examine good practice within and outside of MMU. The first conclusion is that there are no simple answers as to how student assessment workload should be measured and standardised. If there were simple answers then the same practices would be used by most universities. Measuring student, and staff, perceptions of assessment workloads is a complex task and the best that we can hope for is an informed process that attempts to provide consistency within a programme.
Because, in the UK, assessment is measured by the achievement of learning outcomes and not student effort it is probably better to focus on the design of the learning outcomes and their assessment, rather than the direct measurement of effort. Unsurprisingly, students show enormous variation in the effort required to achieve the same outcomes. They also have different perceptions about their workloads which are not linked directly to their actual effort. In one of the few studies of student workload (albeit with a small sample size) Crook and Park (2004) found no significant relationship between the time students spent preparing for assessed coursework and how much that coursework contributed to the module’s overall marks. There was also considerable variation between students, even when the mark achieved was normalised by the effort that they put into the preparation. As Crook and Park (2004) point out:
“it does illustrate one of the difficulties of trying to allocate notional study hours to large and diverse (in terms of module choices) groups of students.”
Since it is impossible to remove these inter-student differences it is difficult, if not impossible, to ensure that all students experience a constant, similar workload. However, it is possible to derive strategies that attempt to provide consistency across a programme such that differences between students are maintained across units. The simplest strategy needed to achieve this aim is the development of programme-specific assessment guidelines.
It is perhaps surprising but, when assessment guidelines are provided, they are remarkably similar. There is an almost universal acceptance that 1 Credit Point = 1 hour of student effort, resulting in 200 hours for a 10 credit unit. Approximately 40-50 hours of these 200 hours are normally allocated to ‘teaching’ leaving 150 – 160 hours to be split between other activities including self-managed learning and assessment. Guidelines differ as to how these non-contact hours should be allocated but the assumed net student workload is generally comparable. For example, if one scheme allocates 50 hours to assessment activities with a ‘norm’ of 10 hours per ‘essay’, this is identical to another scheme which allocates all of the 150 non-contact hours to 5 essay-equivalents.
The essay or report seems to be the base currency for all of the schemes which specify workloads. However, they differ depending if effort is measured in words or hours. It is generally recognised that not all essays are equal and that it is difficult to measure other assessments in essay-equivalent units. Because of the problems of creating word equivalents an assessment guideline based on hours seems to be the most practical measure. Several universities do provide guidelines and they appear to be surprisingly similar. For example, Table 3 compares several schemes in which the student effort has been normalised to the MMU Biology 50 hour baseline and Table 4 shows the total word equivalence for a 20 credit unit from a number of departments or institutions.
Assessment type | Biology (MMU) | EGS (Old scheme) | HLSS (MMU) | Hollings | Clothing, Design and Technology | London South Bank | Bristol |
---|---|---|---|---|---|---|---|
1 hour of examination (including unseen, time-constrained) | 10 |
12.5 |
16.5 |
16.5 |
12.5 |
13 |
|
Seen essay or similar (1500 words) | 10 |
12.5 |
12.5 |
16.5 |
16.5 |
15 |
|
Oral presentation (10 minutes) | 5 |
5 |
7.5 |
7.5 |
|||
Poster (individual) | 10 |
||||||
MCQ (1 hour) | 10 |
||||||
Tutorial (1 hour) | 5 |
||||||
Review (800 words) | 5 |
||||||
Portfolio (6000 words) | 50 |
University (Department) | Total word equivalent |
---|---|
MMU (Biology) | 7500 |
MMU (EGS, Old scheme) | 6000 |
MMU (HLSS) | 6000 |
MMU (Hollings) | 4500 |
MMU (Clothing, Design and Technology) | 4500 |
London South Bank | 4000-5500 |
Leeds (Philosophy) | 4000-6000 |
Latrobe | 5000-6000 |
Sydney | 7700 |
One of the main problems with a time-based workload model is that we cannot rely on the undocumented assumptions of assessors as to the likely time commitment needed from the ‘average’ student to complete a task. As part of the programme guidelines it may be necessary to use ‘auditors’ to review the assumed workload. Alternatively, notional student effort could be monitored by adding a box to the submission proforma which asks students to estimate the length of time spent on the task. Guidelines can then be adjusted in the light of the evidence collected.
Even when guidelines have been developed they should not be over-interpreted because they will never equate to actual hours. Instead they provide a framework which
One key message that comes out of the Open University is the importance of communication between staff and students and between staff. This is best represented by two recommendations from an OU report.
“Communication on the issue of student workload at all levels - university, faculty, course team and to the students is vital.”
“Good pacing and communication plays an essential role in retaining the content that course teams want to include whilst at the same time reducing students feeling overloaded.”
A unit calendar, linked with a stage calendar, is a useful and important tool for guiding students and staff and should help to provide a week-by-week indication of the assumed workload. It may also be helpful if the assessment activities are coordinated, perhaps by the use of ‘assessment weeks’. These would not be reading weeks, instead they would be used for assessment activities. In this way it should be possible to control the student workload more finely whilst also, hopefully, decreasing the time taken to provide marks and feedback.
Crook and Parkes (2004) Measuring assessment: a methodology for investigating undergraduate assessment
Gibbs, G. (2007) Commentary on The Academic Experience of Students in English Universities (2007 report) survey results
(Original Survey)
Entwistle, N. J. and Ramsden, P. 1983. Understanding student learning. Croom Helm, London.
HEA Physical Sciences Centre (2005) Writing Learning Outcomes: Advice on defining courses using an outcomes-based approach
Kember, D. and Leung, D. Y. P. 1998. Influences upon students’ perceptions of workload, Educational Psychology, 18(3): 293–307.
NICATS (2001). Credit guidelines for HE qualifications in England, Wales and Northern Ireland. Guidelines jointly prepared by CQFW (Credit and Qualification Framework for Wales Project), NICATS (Northern Ireland Credit Accumulation and Transfer System), NUCCAT (Northern Universities Consortium for Credit Accumulation and Transfer) and SEEC (Southern England Consortium for Credit Accumulation and Transfer)
Trigwell, K. and Prosser, M. 1991. Improving the quality of student learning: the influence of learning context and student approaches to learning on learning outcomes, Higher Education, 22: 251–266.
e-mail: a.fielding@mmu.ac.uk
telephone: 0161 247 1198
Download this article as a .pdf file
Autumn 2008
ISSN 1477-1241