Using Students' Assignments to Create a Library of Re-usable Learning Objects

Introduction

This paper reports an on-going action research project. The main activity of the project centres on student-produced video in the context of a final level elective: ‘Human Factors for Business Information Systems (HFBIS)’ which runs across the MMU Business School and which focuses on applied psychology operating in a technological domain. Increased availability of video production tools (both software and hardware), plus an increased profile of video podcasting were instrumental in triggering the described change to assessment. This was in a context of a teaching philosophy based on maximising student engagement through:

• Increased student ownership of the assessment process,
• Increased variety of learning experience, and
• Increased opportunities for being creative.

The availability of the technology, in the context of the tutor’s philosophy, led to the design of an assessment based on the production of video, rather than (another) written assignment. As a concept that was new to the Faculty, and so untried, it was offered as an optional route to assessment. This allowed students to produce a written assignment should they feel that this would best suit them. In addition, the video option was only offered for the second of two assignments. This allowed the students to bond with the subject and the tutor, and provided the first assignment as a base-line against which students could assess themselves prior to making the choice.

Video produced for this project had four main purposes:

1. to provide an alternative vehicle for assessment, allowing students to be assessed either by written or video submission,
2. to produce a set of reusable learning objects (RLOs) to support learning of later student cohorts, and thus…
3. to develop an increased sense of a coherent Learning Community, where “this year’s students become teachers to next year’s student”, and
4. to improve student performance as reflected in higher grades and possibly pass rates.

Work on this project commenced with the 2006-2007 cohort, and continues with 2007-2008. As an action research project the methods are evolving. This is a result of modification in the light of experience, and as a result of being able to draw on a library of previously developed RLOs that were not available to the first cohort.

 

Literature

Students experience learning situations in different ways. What they experience operates through at least two mechanisms: i) cognitive predispositions to types of learning (experiential, conceptual, etc. as described by Kolb, 1984); ii) affective: what one finds dull and uninspiring another may find interesting and engaging (paraphrased from Lucas, 2000). This suggests that a more varied approach will have a greater chance of reaching all of the students (at least) some of the time. A blended approach, a mix of on-line, text, experiential educational experiences, fits this requirement.

McFadden and Munns (2002) see the learning situation as a social situation, complete with social relationships (hierarchies, power relationships, etc). By allowing choice (i.e. their choice) about their assessment learners are enabled and empowered. In addition, in the making of a video they may be perceived as establishing a relationship with the students who will follow them. This year’s students become next year’s teachers. The work is explicitly public and to be published. The current publishing mechanism is within the Business School – this meets the teaching and learning need of the Faculty. Consideration has been given to more public arenas, such as youTube and faceBook. However, these are still relatively new, even though popular. It remains to be seen that these virtual locations are problem free with respect to tampering, spamming, and other negative side effects.

Coates (2005) looks at engagement from a managerial perspective. Coates’ view is that any quality assurance (QA) assessment should include measures of students’ engagement. This may be measured by such factors as: attendance, retention, student ratings, student use of facilities for self-directed study, etc. If the work of Lucas, and McFadden and Munns (2002) is appropriate to this context then engagement should improve. In such a situation there would be QA benefits seen as beneficial by management regardless of any other effects.
Pfeiffer et al (2005) propose supporting students in adopting the learning style appropriate to them. Within the HFBIS elective the assignments are considered to be a learning experience, rather than as a snapshot of what has already been learned. Students may submit work by text, or they may take a more visual approach through video. In addition, the video they create requires students to consider planning content, and consideration of audience in a different way than for written work. Video also provides an arena for students to experiment and to explore their creativity. The provision of written and video choices supports different styles. It must be added, however, that the provision of free choice means that students are also free to make the wrong choice. This could be argued to present ethical problems. However, if no choice were to be offered then some students would automatically be disadvantaged (although this would be the choice of the lecturer or the institution).

 

Methods

The project aims to identify the benefits to be gained from assessing students by video rather than written submission. The aim of this paper is to represent the current state of work on the use of student-produced podcasts in the HFBIS elective within the MMU Business School (MMUBS). The project has been running for approximately one year. The terms “video” and “podcast” are used here as follows:

• “video” is what the student works on. They plan the content, take the video, edit it, and submit it for evaluation;
• “podcast” is applied to the usage to be made of the finished and assessed video. The students refer to their work using both terms (video and podcast). A video only becomes a podcast once it is in a distribution mechanism. At present we are using a student-pull mechanism: video is made available on a within-MMU public space for students to download if they so wish. It would be possible to use a push mechanism. In such a case the video would automatically download to the students’ work area without student intervention; however, they would still need to choose to watch it.

Students in the first run of the trial ( the second assignment of the elective for the 06 – 07 cohort ) were allowed to choose to complete a written assignment (six pages plus appendices) or produce a video (c. 5 minutes in length, plus a one page rationale) on the topic of web-site evaluation. Two assignment briefs were written and internally moderated. No significant issues were indicated at that stage. See appendix for details of both the podcast and non-podcast assignment alternatives. The provision of a choice of type of assignment allows students to adopt a style that suits them, and gives them control over more parameters of the assignment. This would fit with the concepts expressed in the literature (e.g. McFadden and Munns, 2002).

During the execution of the assignment students were given the usual support regarding questions of content: tutorial discussion, clarification of concepts, etc. In addition some technical support was available. This was provided in order to allow students a reasonable chance of overcoming technical issues that they had not previously encountered. The main issue here was file conversion. The assignment brief required students to put the finished work into a number of formats. However, this was not a trivial task for some (a result of incompatibility between the hardware and software on their own PCs and those of the university). The provision of QuickTimePro licences, and guidance on use of other software resolved the problems.

 

Findings

The marks students received for the assignment and student feedback on the unit as a whole were analysed. A student with a late submission has been excluded from the analysis to assist with parity of comparison. Two markers were involved, each taking a set of assignments, both written and podcast. Following the marking phase was the moderation phase. The mean marks of the two markers were found to be within two percent of each other. Some further commentary is needed to explain some of the differences between the groups. Within the podcast group there were no failures, and there were no detected cases of plagiarism. Since the student is required to appear in their podcast there is reduced opportunity for plagiarism. In addition, it was the experience of the tutor that students attempting the podcast had higher levels of engagement. This was demonstrated in the questions asked during tutorials and in the creative input to the finished product. It is to be expected that higher levels of engagement would lead to higher marks – one of the objectives of the research; and students with higher levels of engagement may well be more likely to try podcast production. However, this hypothesis has yet to be tested.

Results for the first, written only, assignment are included in table one for comparison purposes. For the purposes of comparison, those found with zero scores in the first assignment have been excluded from the statistics (two cases). For the second assignment there were two non-submissions, so that the count of students is the same for both assignments. The non-submissions of assignment two were not the same students as the plagiarists of assignment one.

Table 1: Summary of marks for written and podcast

	A1: without zeros	A2: whole group	A2: podcasts only	A2: written only
Mean	53.76	56.64	63.15	55.20
Standard deviation	15.48	13.43	11.70	13.44
Min	19	11	45	11
Max	85	86	85	86

 

A student, who had a dyslexia statement of needs, demonstrated a higher level of achievement with the podcast. The first, written assignment was marked at 51. The podcast assignment achieved 62. However, had the podcast assignment brief not required supporting written documentation then the mark would have been higher – there was clear evidence that problems completing the written component cost the student up to eight marks. Thus a brief that was based completely on video submission could, potentially, have raised the achievement of the student from borderline 3rd-2ii up to borderline 2i-1st. No cases of plagiarism were detected during the assessment of either the video or written alternatives.

The mean marks for the written-only first assignment of the elective were reduced by those students who were found to have plagiarised, as demonstrated by TurnitinUK (used to check written assignments). This is an indication of both lower levels of achievement and lower engagement than is desirable. It should also be noted that this issue arose for the first assignment of two – so potential miscreants would be more likely to be deterred for the second assignment, whatever form it took. The requirements of the podcast for the student to appear in it, plus the novelty, or even uniqueness, of the assignment, make plagiarism very difficult.

 

Student feedback

Student feedback, elicited by a standard unit questionnaire was variable, indicating that podcasting worked well for at least some of the students. One student wrote:

“The assignments! Fantastic! Fun!” (sic.)

A balancing comment from another student indicated that they would have liked more support – a not uncommon request (though other students commented that support was good: “superb feedback with support when needed”).

Informally, students did comment that they enjoyed the assignment, but still found it challenging. These challenges were in getting a sufficient grip on the content, working out how to present the content in an appropriate way, and learning the new technology. This last point related to video editing. While many students have experience of taking video few have had to edit with an objective in mind. Editing difficulties related to: special effects (credits, fade, adding a background track, controlling sound levels), and getting to the time limit while maintaining the content.

 

A Learning Community

One aim of this project was to use the developed video to teach later cohorts of students. In this way we get learning benefit from the development of the video, and subsequent learning benefit from those using the video. A recent comment from a (dyslexic) student of the 2007-2008 cohort is as follows:

 

In summary

• The video students had a higher average score: c 6.5% when compared with the whole group, and c8% when compared with written; though there were distorting factors,
• No students failed the video assignment,
• A student known to have dyslexia scored well on the video, but poorly on the written component of that assignment,
• Informal comments on the assessment were positive, though it was not without difficulties
• There is evidence that students find the video to be a useful learning resource and are using them as re-usable learning objects.

 

Discussion

There would seem to be little evidence against the approach – at least from a student perspective. However, there is at least one methodological weakness: parity of assignment and/or evaluation of the work (McLellan, 2004). More work is needed in this area to address various aspects to this problem:

1. Comparability of difficulty of assignment
   1. Content
   2. Length
2. Comparability of marking

The markers both have about 20 years experience as lecturers in Higher Education. This facilitated their work on the setting, marking and moderation of the work as they had developed practice with respect to classifying degree work. For less experienced tutors it would be more necessary to be very careful in writing the assignment brief and marking criteria. Though this is good practice anyway, it is expected that more experienced staff are able to compensate for difficulties in the documentation. External examiners raised no objections to continuation of this approach.

The trial with the first cohort indicated advantages for students in the following areas:

• Support for dyslexic students
• Increased engagement
• Improved retention and progression
• Improved achievement.

There have also been benefits for tutors: greater variation in the style of work being assessed, a certain “entertainment value” in the work. However, these may be confounding factors in any attempt to ensure parity of marking between written and video assignments.
A relatively small proportion of students chose the podcast option: 13 out of 72 (18%). There may be a number of reasons for this. It could be that the most able students chose to podcast. This is explored by comparing the changes in scores for first and second assignments (table 2) and relative means. A further possibility is the extent to which students are prepared to experiment or take risks. This podcast was the first video assignment for these students as part of their degree studies. It was also the first of its type in the Business School – so there were no previous examples for students to use as models for their own work. At present there are no data on how this situation will change when there are examples to refer to, though this will be available at the end of the year.

From table 2. it can be seen that the increases were relatively across all columns. This provides some level of confidence in the process – there was no disproportionate advantage or disadvantage to either form of assignment. The mean increase for video students was 5.31, compared with 2.53 for written. This appears to be a small increase, and based on the small numbers, that may or may not be statistically significant. Should the pattern be repeated over time then confidence in the benefit may increase. Table 3 summarises the results of the first (written only) assignment according to whether or not the second assignment was video or written.

Table 2: Summarised differences between scores for first and second assignments

	Whole group	Podcast only	Written only
Mean	3.03	5.31	2.53
Standard deviation	17.53	15.72	18.00
Minimum	-39	-22	-39
Maximum	42	31	42
Count	70	13	57

 

Table 3: Summary of first assignments for assignment two video and written groups (excluding lates,and zeros)

	Whole group	Written - written	Written - video
Mean	53.76	52.79	57.85
Standard deviation	15.48	15.32	16.71
Minimum	-19	19	37
Maximum	85	84	85
Count	72	58	13

 

Table 3 reveals some interesting points, notwithstanding that statistical significance is in question. The group choosing to produce a video did have a higher average to start with; and that difference widened with the second assignment. The lowest first score for the video group was 37, a borderline fail, for the video assignment the lowest score was 45, a clear pass. Reviewing the raw data showed that this was not the same student improving by 8 marks. Standard deviations and maximums are similar for both groups.

 

Further Work

Further work is needed in this area. Mclellan (2004) identified problems with parity of alternative assignments. This will need to be accounted for in future assignment design, which will require a more rigorous approach to the pre-issuing process.

Formal evaluation is needed in a range of dimensions of student-produced podcasts (indeed, to all educational podcasts). These dimensions include:

• Legibility of text: where text is included the optimum colour / size / typeface need to be considered for a range of formats: full screen down to Apple’s iPod and various mobile phones
• Effectiveness in supporting learning: students seem to enjoy the process and enjoy looking at the podcasts again. However, do they support learning in the context of this elective? Within the MMU Business School, there is evidence from work in the Department of Accounting and Finance (Nick Scott and Keiron Maguire; unpublished work) that accounting students at middle levels of performance gain most from viewing short, skills-based, podcasts. It remains to be seen if this finding is replicated in other subjects and concepts, and for student produced work.
• Support for dyslexia: a more robust statistical study would help to identify the level of benefit (if any) to students with dyslexia. This will require a larger sample of students, and should include markers across a range of units.

Guidelines for presentation (for example on text style, size, etc) will need to be developed and issued to students. This would need to follow from a formal evaluation of optimum characteristics (see formal evaluation above).

Guidelines for structure of content need to be considered. Should students be guided on the construction of a video to meet academic needs? Would this constrain their creativity too much? Should an assignment focusing on learning objectives be supported by a short course in film making? As yet we do not have enough experience to answer these questions. One source of data may be found by identifying those students who come to university with prior experience of media making (e.g. via a media studies ‘A’ Level).

Mobile media players are widely used by students. Are these going to be used for “learning on the move”? If so then the smaller screen size, duration of journey, and a range of other issues need to be identified and included in further studies on the usefulness of podcasts (student produced or otherwise).

A further trial is underway with the 2007 – 2008 cohort. This has continued with the original assignment (modified to remove the written component) and extending the video choice to both elements of the unit assessment (assignments one and two). Results so far indicate that more students will choose to make video for their second assignment having seen more adventurous colleagues use video for the first assignment. The results of the study, so far, indicate that the approach is worth continuing.

References

Lucas,U. (2000), “Worlds Apart: Students’ Experiences of Learning Introductory Accounting”
Critical Perspectives on Accounting 2000 11, 479 - 504

Mcfadden, M. and Munns, G. (2002) “Student Engagement and the Social Relations of Pedagogy”, British Journal of Sociology of Education, Vol. 23, No. 3 (Sep., 2002), pp. 357-366

Coates H. (2005) “The Value of Student Engagement for Higher Education Quality Assurance”, Quality In Higher Education, Vol. 11, No. 1, April 2005

Pfeiffer G., Holley D. and Andrew D. (2005) “Developing Thoughtful Students: Using Learning Styles in an HE Context”, Education + Training Vol. 47 No. 6, 2005 Pp. 422-431 Emerald Group Publishing Limited

Kolb D. A., (1984), “Experiential Learning: experience as the source of learning and development"
Prentice-Hall., New Jersey

Maclellan E., (2004), “How Convincing is Alternative Assessment for Use in Higher Education?” Assessment & Evaluation in Higher Education, 29, 311 - 321

 

Appendix: Assignment briefs for 2006 – 2007

Human Factors for Business Information Systems (pdf doc)