MicroCPD: Computer-Assisted Assessments using Canvas

This week Dr Scott Hayward discusses his experiences of using Canvas for computer-assisted assessments.

E-learning and online resources form the foundation of teaching in higher education, yet the move towards online computer assisted assessments has been much slower.  We set up an online assessment for a 2^nd Yr Biosciences module (BIO259) with the primary goal of seeing if it could make the assessment process more efficient.

Rational:

Paper exams have disadvantages

• Admin load is high
• Marking and feedback are slow
• Not authentic
• Resource heavy (especially staff time, but also use of paper, printers etc.)
• Many opportunities for error of process

On-line exam Advantages

• Efficient
• Authentic
• Fast
• Facilitates feedback
• Inclusive
• No need to transcribe marks
• No need for AV for e.g. RAP students as long as main venue is booked to accommodate extra time
• No need to store hard copies for external examiners etc.

BIO259 Requirements

• Need to have controlled time limited Assessment – can be done in computer clusters. Use of Respondus software to restrict internet outside  of Canvas
• Question formats: SAQ and MCQ  
• Sometimes the question doesn’t lend itself well to Canvas and had to be modified, e.g. drawing diagrams
• Questions were drawn from a question bank and order of questions needed to be controlled
• Students needed to be able to see all questions (replicating the ability to go back and modify in a paper test)
• Students were allowed to see the MCQ scores once the test was submitted

The test

• Students were introduced to the Respondus software and logged on
• Students were instructed to start together
• Some paper copies were available in case of computer failure but these were not needed
• No significant problem for students in the venue were reported or afterwards – there were a couple of students who initially had trouble logging on, but this was quickly resolved.  Highlights requirement for IT support at venue.

Marking

• The CSV file was downloaded with each students answer on a single line
• Two extra columns were inserted after each SAQ question for markers comments and the mark
• Spreadsheet was emailed to the markers
• Completed marks were pasted into the master spreadsheet
• Final test marks of combined =MCQ and SAQ calculated by formula
• Marks returned to office
• SAQ Feedback returned to students as a combination of generic and specific via mail merge

Feedback from the Teaching Office:

• Simple and easy method of administration.
• Allows more effective and efficient use of administrator time to be spent on other areas of role and responsibilities.
• Accurate information obtained through online system – reduced risk of human error.
• The online only submission is easier for students and admin staff.

Feedback from students:

Positives

• typing SAQ's is much faster than doing it in a written format.
• I can clearly see what I am doing on the online format, it is much easier to go back and change things rather than crossing out paragraphs, running out of space, asking for paper etc.
• personally I found it easier to articulate myself 
• As I type up my notes all year I feel far more confident answering questions online as I think I can type faster than I can write
• I think the online format should be the ideal format for exams moving forward

Issues

• There needs to be an option to undo a selected answer if MCQs are negatively marked.
• Not sure I liked seeing my MCQ mark straight after submitting it as people ask each other what they got as soon as you come out of the test.
• Answering SAQs in this way is that you cannot always use a diagram. 

Factors to consider

Set-up – checking the quiz runs and appears in the way designed, including images loading correctly, question formats being correct etc.

On the day – there will always be the risk of problems, but generally these can be overcome quite easily with Digital team on hand and paper copies of test as backup; e.g. one student inadvertently caught the on/off switch and rebooted their computer when trying to move the screen. The student was able to log back into canvas and continue from where he’d left off.

The LES Birmingham Digital Education team have a one page handout available for staff on how to moderate a live exam, detailing how to provide students with extra time or attempts if required. In addition to this we have simple to follow instructions for students on how to launch the locked down browser.

The HEFi Gateway also has detailed instructions:  https://canvas.bham.ac.uk/courses/18055/pages/about-unit-8-using-respondus-lockdown-lab

Post Exam – Once the exam is completed by the student, they will see an initial mark, though this is as yet unmoderated by the examiners and will not include marks from any SAQs  - this needs reviewing. Teaching staff can download the student results as a CSV file to be integrated and marked potentially in Excel.

Further Reading

HEA resource for computer assisted assessment: https://www.heacademy.ac.uk/knowledge-hub/computer-assisted-assessment-caa