An evaluation of enhanced student support (including podcasting) on assessed course work achievement and student satisfaction
“I can play you, fast forward you, rewind you and when I’m bored of you I turn you off”.
My heart sank as an initial reaction to listening to this feedback about my course podcasting from a student. On calm reflection, it’s a real compliment and it reflects the learning style of many students. Podcasting transfers control of the learning process to the student and provides support to students when they want it and not during timetabled sessions or when staff are available.
My interest in podcasting began in Autumn 2006 when my daughter brought home a computer memory stick from school, plugged it into her PC and completed an aural French homework exercise. It became obvious that this was effective. She worked at her own pace and it only required technology that is freely available to all students. Following some experimentation with audio podcasts I submitted a joint application with Kieran Maguire for a Fellowship in Academic Practice in December 2006 to develop the use of audio and video podcasting within our teaching practice. Neither of us could have anticipated the results or the enthusiasm with which students have embraced it.
This short article summarises the evaluation of one of those projects. However, it’s not about podcasting alone!
Advanced Accounting Theory and Practice (ATP2) is a final year core unit for BA Accounting & Finance students. A prerequisite for the course is having studied Accounting Theory and Practice (ATP) in the second year of their studies. This unit is taught by Kieran Maguire and over the last three years attempts have been made to align more closely the detailed course content of both modules. The syllabus and assessment is influenced by the professional accountancy bodies (principally ACCA) in order to obtain exemptions from future studies for students.
Student numbers have doubled to over 220 in the last 5 years as a result of the successful positioning of the course in the MMUBS portfolio. This provided a number of strategic challenges that are summarised in Figure 1 below.
Figure 1: Some of the challenges facing the course team
The course team recognised that literacy and critical thinking skills needed developing and a flexible approach to learning was needed to address the changing lifestyles of students. One issue particularly noted by the course team was christened the “September Effect”. This reflects the introduction of students to skills training (such as literacy skills) in prior academic years or at the start of academic courses (in September) which need reinforcing later in the year to ensure the students can apply them to a current situation.
Figure 2: The model adopted to address the identified learning issues
To address the learning issues identified the model in Figure 2 was adopted and this incorporated:
- The use of course packs. Course packs (Figure 3) are a pre-printed bound set of material for the students to use over the year with web addresses for resources, websites and library e-journals.
- Embedded Information Literacy – creating reading lists for each week of the course and copying Copyright Licensing Agency (CLA) cleared versions of relevant literature into the course pack.
- Use of a range of tutorials – alternate week critical thinking tutorials were introduced to improve students’ critical analysis skills. In effect students obtained 2.5 hours of timetabled contact time per week with a trade-off of larger class sizes for two of those hours
- Introduction of WebCT Vista – in such a way as to encourage student reading by careful consideration of the structure of this resource
- A structured approach to podcasting complementary materials for students and NOT materials that substitute for contact hours
- E-reading - the digitisation of a number of core book chapters in PDF format that were included on the library reading list and on WebCT Vista
- Out of hours support – all student e-mails to be answered within 24 hours (including weekends and vacations) at critical times such as assessment submission dates.
The red ovals are the skills set required by students to succeed in final year studies. They arise primarily from prior learning. However, these are often weak areas that let students down in their final year and need continual reinforcement.
The major change was the introduction of audio and video podcasting. Pedagogical models for podcasting are in their infancy. Edirisingha and Salmon (2007) have reviewed the work in progress of a national study called Informal Mobile Podcasting And Learning Adaptation (IMPALA). Their paper discusses the assistance of podcasting techniques for managing the learning of large student cohorts, particularly where podcast video techniques can be used to replace paper based hand outs that would need screen shots and graphical illustrations. The paper also raises a number of interesting issues in course design when using podcasting.
Podcasting was used extensively to support the assessed coursework process. The coursework comprises a traditional 2,000 word assessed course essay on a topical corporate reporting issue. It accounts for 30% of the unit mark. The support of the assessed course work was designed to provide greater support to students, improving the student experience and addressing the key issues of student achievement and progression.
Students were supported on their coursework in the ways shown in Table 1.
|Initial Lecture||Traditional briefing lecture covering the topic, key areas, key requirements and a demonstration of research techniques (including library e-resources). The aim of the research techniques was to address the ‘Googling’ approach to research.|
|E-mail support||Maximum 24 hour turnaround of individual e-mail questions from individual students including weekends.|
|WebCT resources||Some basic resources such as websites, hints & tips, basic reading list and other items were all contained in one folder on WebCT.|
|WebCT mail||Use of all student mail to deal with common issues and to brief \ update whole student group. 17 messages were delivered to all students.|
|Podcasts||A number of podcasts were used to support students. See details below.|
|Referencing and plagiarism||Links to relevant library sites and a copy of the MMUBS plagiarism student briefing document were provided with an introductory discussion.|
|MMU Hours||Student ‘open’ drop in sessions with tutors were increased to 4 available hours per week in the 2 weeks before the assessment due date|
|Minshull drop-in||Two separate sessions were held as ‘drop-in’ arrangements two weeks before the hand in date. Students were advised that a short presentation would be made of FAQs together with a question and answer session. These sessions were recorded and podcasts made available to students.|
|Enhanced feedback||The standard MMUBS feedback form was amended to demonstrate to students how their work addressed some of the key points in the initial briefing using a tick box approach. Feedback scores were returned within 2 weeks of the hand in date.|
The major innovation was the use of audio and video podcasts to support students. These consisted of:
Library skills podcasts: These library skills related video podcasts demonstrated how to search relevant websites and to use electronic resources on the library website. They were prepared by doing live internet searches and recording the screen with commentary as they were undertaken. Download an example clip.
Reinforcement podcasts: A video podcast that recapped the main points from the assignment briefing using PowerPoint slides and other electronic resources. This recapped some key slides from the original lecture briefing and covered some new areas that had arisen from student questions. Download an example clip.
Referencing and plagiarism: A study skills video podcast covering referencing and plagiarism prepared by Emma Flynn (MMUBS Student Support Officer). This is a podcast version of the study skills session which students can attend voluntarily. It is based around Podcasting annotated PowerPoint slides on a Tablet PC. Download the podcast.
Technical support podcasts: Accounting & finance specific video podcasts that are based on an introduction to some of the technical material required.
FAQ audio podcasts: MP3 audio podcasts recorded live during question and answer sessions held with the students and published after the sessions.
Seven podcasts were produced in all and stored on an MMU server (pod.mmu.ac.uk). Links were created to each from WebCT Vista. Students could download the podcasts to their own PC or click on the WebCT links and play by streaming them. Students could replay them as often and whenever they wanted so overcoming the ‘September effect’.
The podcasts were developed in a student centred manner by adopting a constructivist approach with a focus on the knowledge construction of the learner. Gagnon (2007) suggests that teachers should focus on how to organise what learners will do. The role of the podcasts is as a cognitive guide rather than knowledge transfer. Hawryszkiewycz (2007) recognizes that there is a need to assist learners in constructing knowledge by providing them with customised online learning environments for various different learning demands. In this case a range of customised podcasts was produced to meet the learning needs of the student cohort. Students were free to pick and choose which resources they needed based upon their own learning needs.
A detailed quantitative and qualitative marking and moderation process was undertaken. The results, with prior year comparatives, are summarised in Table 2 below:
The more interesting analysis is given in Table 3 of the grade distribution compared with the previous year.
There was a marked increase in the proportion of students obtaining a first or upper second class degree. This confirmed the view of the unit leader that the mid stream students had benefited most from the additional support. The assessment curve had flattened. The proportion of students achieving a first or upper second degree classification mark increased from 40.1% to 51.0%.
Student achievement in assessed course work can be significantly enhanced through the use of improved assessment support. In particular the use of podcasts had a significant effect in improving basic research and referencing skills.
Use of Podcasts Provided
All of the additional materials were delivered through WebCT. WebCT provides detailed student tracking activity. In effect it records every student ‘click’ when they are within the WebCT environment. It provides a comprehensive analysis of student activity. One limitation is that it cannot differentiate between saving a file to disk or reading / using it online. For example, if a student clicks on a podcast link they can either save it to their own hard drive or play it by using the default media player. Therefore one click is recorded. If the student saves the podcast we have no subsequent detailed record of how often the student used the podcast offline. The same applies to any file type that is accessible on WebCT. However, it does provide some indicative evidence of student use.
In total the seven podcasts were accessed 1,993 times by the 214 students (average 9 times per student). This is high given that many students download it (one click) and watch it offline. Of interest are the student work patterns. The number of clicks increases significantly towards the hand in date for the assessed work. Over 50% of all clicks occurred in the week prior to the hand in date and 25% of all clicks occurred on the hand in date itself.
Figure 4 shows the number of clicks on the podcasts per week over the five week period between the assignment briefing and the hand in date. WebCT statistics show 498 accesses of the podcasts on the hand in date alone.
Figure 4: Clicks on WebCT per week
The research skills podcast was most popular. The students used it throughout the process. The reinforcement podcast was similarly accessed extensively. It is notable that the referencing and plagiarism podcast was accessed more frequently in the week leading up to the hand in date. This podcast was obviously more important to the writing up process. However, it had far fewer accesses than the other two main podcasts. This may reflect the students’ perception of importance.
The usage of these podcasts reflects the tutors’ feelings that:
- the quality of research and the range of sources cited by students was much better than in previous years
- the quality of referencing (using the Harvard system) had improved but there was still a significant minority of students who had not addressed this
One conclusion is that MMUBS students would benefit from a range of ‘generic’ skills podcasts made available through WebCT covering:
- library skills – research, resource access and literacy skills
- study skills podcasts – plagiarism, essay writing etc.
However, these standing resources would need to be advertised and embraced by unit leaders for their courses. Students would use these.
One student commented: “Maybe a general university podcast on how to write a good essay could help”.
The other podcasts were all well used by students and were of value. These were subject specific podcasts. They undoubtedly contributed to student achievement and the resources need to be made available to allow staff to create and share their own podcasts.
The student usage results may provide evidence of the use of podcasts towards the end of the previous academic year. These were hosted on social networking sites such as MySpace. They were designed as question practice revision aids. The counters on those sites showed that students were on average watching the podcasts three times per day in the week leading up to the examination. The confirmation of the examination support approach will be tested later in the year.
There is no evidence that podcasting course resources have had an adverse effect on student attendance. The initial lecture was attended by an estimate of 90% of the registered students and the podcasts provided support for these students and a number of other students who enrolled late and missed a number of the early sessions. A number of students have remarked that the podcasts have supported their studies where they have missed sessions due to late enrolment, illness or work commitments.
Use of other materials provided
The students used all of the materials on WebCT, confirming that this system is an effective method of directing student learning. Of particular interest is a comparison of the usage of a word document provided covering plagiarism and referencing and the equivalent podcast. The students used the podcast resource over 2.5 times more frequently than the paper equivalent.
Use of resources by low scoring students
The enhanced support had no apparent effect on students who achieved a mark of lower than 50%. The WebCT statistics of these students showed that the use of the podcast resources by these students was approximately 60% of those of an average student. None of those students had contacted the unit leader with queries and almost half of them had not read any of the e-mails sent out by the course team. These students had made little use of the enhanced resources despite their extensive promotion.
Student Feedback – Student experience andstudent satisfaction
Student feedback was sought in a number of ways including informal feedback to the tutor team, a formal class based survey, student voice podcast feedback and e-mail comments. Around 60% of students completed a comprehensive survey of the support received. Of these:
- 98% had watched podcasts and found them useful
- 98% would recommend podcast support to be used on other units and courses
- 81% watched the podcasts more than once
- 56% downloaded the podcasts onto their own PC
- 98% want more podcasts on the ATP2 course
- 71% accessed the podcasts at home, 14% at MMU and 15% at both
The statistics confirm the overwhelming success of the podcasts. The accessing of podcasts at home by students confirms the need for MMUBS to have a robust, accessible web based presence for podcast upload and delivery.
Students scored the overall support they received on this piece of assessed course work at 9.3 out of 10 (93%).
The students rated the effectiveness of podcast support for this assignment as 9.5 out of 10 (95%).
Caution is needed in that students do not bear the cost of producing podcasts or additional materials and so may therefore request additional resources without needing to consider the implications. However, the high levels of usage, student satisfaction and improved student attainment all justify a widening of the use of podcast and enhanced support.
All of the students’ comments were supportive such as
“This (Podcasting) is the best thing happening at university”.
“The podcasts are excellent for exam revision and assignment support”
“It is a huge advantage for foreign students where English is not their first language as they can replay points they didn’t pick up in lectures or tutorials”
“It would be great if all lecturers produced them”
“Having missed one of the assignment sessions due to illness normally I would have missed out, but I could listen to it at home and still get the info”
“The podcasts improved my ability to research and helped me to reference my work”
A number of students provided voice podcast feedback in a ‘podbooth’. They dictated their comments into a digital voice recorder. Their unedited comments can be accessed from this link.
Kirkwood (2003) claims that: “Independent learners value the richness and flexibility they derive from having a range of media components in their courses”. The feedback received from MMUBS students confirms this.
The Tutor’s View
Podcasting involves some personal upfront investment in the process but it should not be viewed in isolation from the whole student support process. The initial investment in podcasts could be reduced by having a generic set of library and skills podcasts. Lecturers would then only need to prepare subject specific resources. This can often be achieved by recording sessions live such as the assignment briefing.
Production quality of the podcasts does not appear to be an issue with students. Only the basic facilities of Podcasting software have been used. Enhanced features such as picture in picture and advanced editing can be undertaken. However, students are happy with ‘one take’ podcasts with limited editing and this is quick for staff to master.
The pressure on the lecturer is undoubtedly reduced. Podcasting is an efficient method to communicate with all students. It undoubtedly reduced the number of contacts initiated by students (face to face or e-mail) and is therefore a great aid to efficiency.
This is confirmed by the student comments on their questionnaire forms. For example when commenting on why they did not contact the tutor during the process comments included:
“Got enough information from podcasts”
“Enough answers were given to support me. So I didn’t need to e-mail Nick”
“All the podcasts provided gave me all the information I needed for preparation of the assignment. If I am not sure about something, all I do is replay the podcast and pause in between to make sure I completely understand what is being said”
“All the questions I wanted to ask were answered in podcasts”
“I was ill when the assessment sessions were on in the Minshull building so (I) listened to the voice podcast on WebCT and all my questions were answered so I didn’t have to see or e-mail Nick”
The students certainly benefited from the search skills and referencing guidance podcasts, which showed significant improvements in achievement from prior student cohorts.
The approach taken to podcasting to date has been to develop complementary resources rather than substituting for existing contact time. Students however have lobbied for the podcasting of lectures and tutorials. They cite reasons such as absence through illness, family issues, students with children responsibilities, English being their second language and difficulties understanding lecturers. I have changed my own opinions of this over the last few weeks and I think there are valid reasons for podcasting lectures and tutorials. The key issue is whether it has an adverse effect on attendance and whether this matters. I have produced podcasts of tutorials and to date it doesn’t appear to have affected attendance. However, this is a final year unit and the effect in earlier years may be different.
Podcasting is an effective method of reinforcing knowledge and skills. It is a flexible method that allows the student to plan and control their learning. Students work patterns do not usually mirror the lecturer’s availability and podcasting provides valuable support in ‘out of hours’ situations. It eliminates the need to repeat information to different groups of students and improves lecturer efficiency. However, it is essential that podcasting is considered in the course design process and not simply bolted on to existing programs. Most importantly the initial evidence suggests that it meets three important criteria:
- it improves the student experience and student satisfaction
- it has a positive impact on student attainment
- it can assist in improving student progression
However, this is a very early and tentative conclusion. EDUCAUSE (2008) suggests the effects of podcasting on students in higher education have yet to be fully examined.
Where from here?
Colleagues in MMUBS are now developing podcasts to support students during the revision and final examination process. A similar student survey and results analysis will be undertaken. This use of podcasting for information literacy is developing rapidly. YouTube (2008) contains many video clips on how to access resources.
Nick is available to demonstrate the range of podcasting used in the Division of Accounting & Finance to other departments and faculties.
Nick would like to acknowledge the contributions of Emma Flynn, Kieran Maguire and Graham Holt in supporting the assessed work process for this unit. In addition, David Matthews and the library staff at MMUBS provided valuable assistance in digitising copyright cleared documents so that students had electronic access to all of the course resources. Additional information literacy support ideas were provided by Clare Scott (University of Sheffield).
Nick would also like to thank Alan Greenberg at Apple for his continuing support and providing a number of course prizes for students.
Edirisingha, P. & Salmon, G. (2007). Pedagogical models for podcasts in higher education. Retrieved on 1 February 2008
EDUCAUSE (2008) The Horizon Report, 2006 Edition, Austin, Tx: The New Media Consortium and Boulder, Co: Educause Learning Initiative, Retrieved on 3 February 2008
Gagnon, G. & Collay M. (2008) Constructivist Learning Design. Retrieved 1 February 2008
Hawryszkiewycz, I. & Pan, W. (2007) Facilitating Knowledge Construction by Customised Supportive Services, Journal of Education and Information Technologies, 12, 245-265
Kirkwood, A. (2003) Understanding Independent Learners’ Use of Media Technologies, Open Learning: The Journal of Open and Distance Learning, 18:2, 155-175.
YouTube (2007) Using Google Scholar, Retrieved 3 February 2008