Evaluation of undergraduate ophthalmology medical students' methods of assessment during COVID-19 pandemic

INTRODUCTION

Most educators agree that there are 2 main types of assessment criteria, formative examinations (FE) which occur continuously during the educational process and provide systematic continuous feed-back at all stages of the educational situations, ¹ and summative assessment which occurs at the end of the learning process resulting in a ranking, a mark or a degree.²

Multiple choice questions (MCQ) can assess factual recall and problem-solving.³ They are often used to test student’s knowledge of a broad range of content and are preferred for their objectivity and ease of scoring a large bulk of students at a time. Correct framing of MCQ is thus essential.⁴ Concerns have been raised about MCQ limitations. These include cueing. Question construction is also a concern; both poor grammar and question structure reduce test validity.⁵ Critics of MCQ examination have also raised  concerns  about  whether  such  questions  can assess the higher cognitive levels and the clinical skills needed for professional success.⁶

Short answer questions (SAQ) are most often used to test basic knowledge of key facts and terms. SAQ can also be used to test higher thinking skills, including analysis or evaluation.⁷ SAQ can be constructed faster than MCQ.⁶ Unlike MCQ, SAQ make it difficult for students to guess the answer. SAQ provide students with more flexibility to explain their understanding and demonstrate creativity than they would have with MCQ; this also means that scoring is relatively laborious and can be quite subjective.⁸ SAQ provide more structure than essay questions (EQ) and thus are often easy and faster to mark and often test a broader range of the course content than full EQ.⁷

Like SAQ, EQ provide students with an opportunity to explain their understanding and demonstrate creativity.⁸ EQ make it hard for students to arrive at an acceptable answer by bluffing. EQ can be constructed reasonably quickly and easily but the use of only essay questions restricts the overall scope of an exam to a few topics or areas. Marking these questions can be time-consuming and grader agreement can be difficult.⁹

Case scenario based multiple choice questions (CS-MCQ) provide opportunities for the integration of sub-specialties and assessment in keeping with problem based learning.¹⁰ CS-MCQ are reliable and practical and promote multi-logical and critical thinking.¹¹

Oral exams (OE) allow students to respond directly to the instructor’s questions and/or to present prepared statements. These exams are especially popular in language courses that demand ‘speaking’ but they can be used to assess understanding in almost any course by following the guidelines for the composition of SAQ.¹² Advantages of OE include that they provide nearly immediate feedback and so allow the student to learn as they are tested. There are drawbacks to OE including the amount of time required and the problem of record-keeping.¹³

Corona virus disease 2019 (COVID-19) lockdown had negatively impacted the methods of face to face learning and practical on-patient methods of students' assessment.¹⁴ COVID-19 forced a transformation from traditional face-to-face learning to electronic learning,¹⁵ and that this transformation will bring long-lasting effects on teaching, learning, assessment procedures and methods.¹⁶ So, the aim of this work is to evaluate the methods of undergraduate ophthalmology students' assessment during the integrated ophthalmology learning program of Al-Azhar Faculty of Medicine of Boys (Cairo,2021) by using variable forms of examinations and items (questions) in absence of face to face OE and on-patient practical examinations (PE) during the COVID-19 pandemic. And to answer the question: Can using of variable forms of assessment tools including objective structured clinical examinations (OSCE), MCQ, SAQ, EQ and CS based questions replace OE and on-patient PE for undergraduate ophthalmology medical students?

MATERIALS AND METHODS

230 third level students responded to survey of 24 Likert Scale questionnaires (Q) using Google platform to evaluate the methods used for students' assessment after completing 6 weeks of integrated ophthalmology course (from 16^th March, 2021 to 22^nd April, 2021). The students' assessment during the course included three online FE (Two single best answer (SBA) exams and one modified OSCE), work sheet activity, summative SBA exam and OSCE at the end of the course, and final summative written exam at the end of semester including single best answer questions (SBAQ), SAQ, structured EQ and CS based questions. The FE were done using Google platforms, and the summative OSCE and written exams were done under complete protective and social distancing situations due to COVID-19 partial lockdown in Egypt. The total marks were 100 and the students' assessment did not include OE or live practical on-patient exams.

The analysis of students' response was based on using five degrees points from 1 (strongly disagree) to 5 (strongly agree). Students were assured of their confidentiality and were promised that their names would not appear in the document.

The evaluation of methods of undergraduate ophthalmology students' assessment was also done by analyzing the total marks achieved by each student at the end of semester. 

Statistical package for the social sciences (SPSS) program version 24 (IBM corporation, New York) was used. Mode, mean (M) and standard deviation (SD) values were calculated, paired t-test was used to measure the statistical difference between students' response  from neutrality and probability (P) value was significant if P < 0.05.

RESULTS

The participant students responded to all Q (figures 1-24) and statistical data of students' responses (table 1) including mode, M, SD, t-test results and P value were analyzed.

61 % of students (20 % strongly agreed and 41 % agreed) reported that FE helped them for the following summative practical OSCE and written exams, while 28 % of students had neutral response and 11 % of them disagreed. In addition, 48 % of students agreed that three FE during the course were enough while 22 % of them disagreed.

Most students reported that the end semester examination paper was regular, all question sections were clearly printed and the marks distribution was clear. Most students preferred the use of MCQ with 4 options rather than MCQ with 5 options. Most students reported that the duration of summative OSCE was suitable. 46 % of students agreed that the duration of end-module SBA exam was suitable while 20 % of them disagreed. 47 % of students agreed that the duration of end-semester written exam was suitable while 23 % of them disagreed.

Most students reported that OSCE slides (figure 25) covered the assessment of most studied ophthalmology signs with high quality images. 49 % of students agreed that data-show dependent OSCE can replace live on-patient PE during corona pandemic while 19% of them disagreed. On asking about the possibility of replacing on-patient PE with OSCE for all times; 42 % of students agreed but the responses were not conclusive and were not statistically significant different from neutrality.

Most students reported that the variation of question forms allowed assessment of most course topics precisely and simply, so that student should review and study all chapters and topics preparing for the exam and they should not only focus on famous topics to pass the exam.

The majority of students (72 %) agreed that using variable forms of questions including MCQ, SAQ, structured limited EQ and CS based questions can always replace OE for undergraduate ophthalmology students' assessment and this was safer than face to face OE during corona pandemic. The majority of students (70 %) reported that using variable forms of written questions can save time more than OE. 68 % of students reported that marks can be more easily and correctly calculated with more justification by using variable forms of written questions than OE. 62 % of students reported that using variable forms of written questions are less suspected to human judging factors and variability of questions difficulty levels than OE. By asking about OE can give more self confidence and practical sense to the students, the responses were not conclusive or obviously different from neutrality.

 Most students agreed that CS based MCQ and SAQ can evaluate higher degrees of students thinking levels including application, evaluation and analysis. 44% of students agreed that CS based questions can help to replace live on-patient PE for undergraduate ophthalmology students during corona pandemic. The student responses were not statistically different from neutrality on asking if they prefer CS questions and if they have trained enough to answer these types of questions.

Analysis of total marks achieved by each student (table 2) showed that 97.8 % of students had passed the ophthalmology module examinations with 44.2 % of them had A+ values (≥ 90 marks).

Fig. 1: Students' responses for Q1 (FE helped me for the following summative OSCE and written exams.)

Fig. 2: Students' responses for Q2 (FE were enough in number during the course.)

Fig. 3: Students' responses for Q3 (The end-semester examination paper was regular, all question sections were clearly printed and the marks distribution was clear.)

Fig. 4: Students' responses for Q4 (I prefer MCQ with 4 options than MCQ with 5 options.)

Fig. 5: Students' responses for Q5 (The duration of summative OSCE was suitable.)

Fig. 6: Students' responses for Q6 (The duration of end-module written exam was suitable.) 

Fig. 7: Students' responses for Q7 (The duration of end-semester written exam was suitable.)

Fig. 8: Students' responses for Q8 (The summative OSCE covered the assessment of most studied ophthalmology signs with high quality images.)

Fig. 9: Students' responses for Q9 (Data-show images in OSCE can replace live patient dependent PE for undergraduate ophthalmology student assessment during corona pandemic.)

Fig. 10: Students' responses for Q10 (Data-show images in OSCE can always replace live patient dependent PE for undergraduate ophthalmology student assessment.)

Fig. 11: Students' responses for Q11 (The variation of question forms allows assessment of most course topics precisely and simply.) 

Fig 12: Students' responses for Q12 (With integrated ophthalmology course; I should read and study all topics before the exam and I should not focus on famous topics to pass the exam.)

Fig. 13: Students' responses for Q13 (Use of variable forms of questions like MCQ, SAQ, structured limited short EQ and CS based questions can replace OE.) 

Fig. 14: Students' responses for Q14 (Written assessment with variable forms of questions like MCQ, SAQ and structured limited EQ were safer than face to face OE during corona pandemic.)

Fig. 15: Students' responses for Q15 (Written assessment with variable forms of questions like MCQ, SAQ and structured limited EQ can save time more than OE.)

Fig. 16: Students' responses for Q16 (In written assessment including variable forms of questions; the marks can be easily and correctly calculated with more justification than OE.)

Fig. 17: Students' responses for Q17 (Written assessment using MCQ, SAQ and short EQ is less suspected to human judging factors and variability in questions difficulties than OE.)

Fig. 18: Students' responses for Q18 (OE provide the students with more self confidence.)

Fig. 19: Students' responses for Q19 (OE provide the students with more practical sense.)

Fig. 20: Students' responses for Q20 (CS based MCQ and SAQ can evaluate higher degrees of student thinking and cognitive levels like application, evaluation and analysis.)

Fig. 21: Students' responses for Q21 (CS based MCQ and SAQ can always replace live patient dependent PE and live history taking skills in ophthalmology.)

Fig. 22: Students' responses for Q22 (CS based MCQ and SAQ can replace live patient dependent PE in ophthalmology during corona pandemic.)

Fig. 23: Students' responses for Q23 (I prefer CS based questions than traditional direct questions.)

Fig. 24: Students' responses for Q24 (I have trained enough to answer CS based questions.)

Fig. 25: Examples for OSCE slides

Table 1: Statistical analysis of students' responses

Table 2:  Analysis of total marks achieved by each student