Evaluation research in Korean medical education: a systematic review

Article information

J Yeungnam Med Sci. 2023;40(3):233-240

Publication date (electronic) : 2022 December 5

doi : https://doi.org/10.12701/jyms.2022.00563

Hye Jin Park¹, Yu Ra Kim^,2

¹Department of Medical Education, Eulji University School of Medicine, Daejeon, Korea

²Department of Medical Humanities, Yeungnam University College of Medicine, Daegu, Korea

Corresponding author: Yu Ra Kim, PhD Department of Medical Humanities, Yeungnam University College of Medicine, 170 Hyeonchung-ro, Nam-gu, Daegu 42415, Korea Tel: +82-53-640-6990 • Fax: +82-53-629-2252 • E-mail: yurakim@yu.ac.kr

Received 2022 August 19; Revised 2022 September 16; Accepted 2022 September 23.

Abstract

Introduction

In Korean education, evaluation was mainly used to describe a test for students in schools [1]. However, the meaning and role of evaluation has been expanding recently because of increased interest in evaluation for academic characteristics. Thus, institutions in charge of the professional evaluation of educational programs have been established, and the purpose of these institutions is the quality management and certification of educational programs [1,2]. Because Korean medical education focuses on the transfer of knowledge and student academic achievement is judged only by intellectual abilities [3], evaluation targets are often limited to students. However, interest in the role or meaning of evaluation in medical education increased as the Korean Institute of Medical Education and Evaluation (KIMEE) played a role in managing the quality of medical education and accrediting medical schools.

According to the development history of education evaluation outlined by Lee [4], education evaluation began to be known as an independent discipline in the early 20th century, and the meaning of education evaluation expanded and changed as follows. While early educational evaluation focused on measuring characteristics of the subject being evaluated, second-generation educational assessment focused on comparisons between the actual data obtained and the description of the behavior a student exhibits when successfully completing a curriculum or program. Third-generation educational evaluation focused on experts making professional judgments on the evaluation target. Fourth-generation educational evaluation focused on the role of the evaluator in responding to the needs of stakeholders, gathering agendas to be managed in the negotiation process from a phenomenological perspective, providing information needed by the stakeholders, and guiding the adjustment of opinions.

With the introduction of performance-based education aimed at complete learning and norm-oriented evaluation in medical education [5], it is inevitable that educational evaluation activities take place throughout the course of medical education. (1) Before class: are the process outcome establishment and class design suitable? (2) In class: are students achieving the learning objectives? If not, what improvements can be undertaken? If objectives are being achieved, how can further improvements be implemented? (3) After class: students, subject, curriculum, and overall evaluations of medical school education are implemented. Evaluation studies of Korean medical education are ongoing; therefore, researchers in the field intend to analyze how evaluation studies are being conducted in Korean medical education.

The purpose of this study was to collect basic data for research on the improvement in medical education through evaluation because the meaning and role of evaluation has expanded in Korea after the introduction of the performance-based paradigm, evaluation, and certification system in medical education. In this regard, this study aimed to identify research trends in the evaluation of Korean medical education in the 21st century.

Design

2. Selection and exclusion criteria

The selection criteria for the publications extracted in this study were as follows. First, the keywords were limited to Korea, medical education, and evaluation, and the presentation period was not limited. Second, this study was aimed at academic journals (except for candidate papers, research reports, posters, conference presentations, books, internet materials, and other types of research) of Korea Citation Index (KCI)-registered (candidate) journals or higher. Third, only subjects that were confirmed to be originally written in Korean were included. Fourth, the subjects of the evaluation study were limited to students associated with basic medical education (BME) and medical schools, and data from major doctors and patients were excluded.

The research problem of this study was the analysis of research trends, subjects, and results regarding the evaluation of Korean BME. To answer this research question, two analysts reviewed and discussed the collection and selection of literature, derivation of analysis criteria, and coding and analysis of results. This study was conducted in the following manner according to the standards of systematic review suggested by Cook et al. [6].

First, the bias and subjectivity of the study were removed based on PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [7]. Second, the research issues were clarified by analyzing the trends, subjects, and results of evaluation research in Korean BME. Third, the literature search, literature selection, analysis results, and discussion were conducted to draw a comprehensive conclusion by selecting appropriate literature and analyzing its research content.

As a result of searching the database according to the scope and criteria of our analysis, 5,205 papers were identified, 3,199 papers were selected through the first screening process, 242 papers were selected through the second screening process, and 143 papers were selected through the final selection process (Fig. 1, Supplementary material 1).

Fig. 1.

Strategy of the scoping review. KCI, Korea Citation Index.

Literature related to evaluation

The research team reviewed 143 papers that were extracted based on evaluation of Korean medical education and classified them into three categories according to the subject of evaluation: student evaluation, program evaluation, and accreditation (Table 1). The literature was classified as follows. (1) Student evaluation: literature that evaluated or researched student-learning abilities. (2) Program evaluation: literature that evaluated or researched educational programs, including subjects or curricula. (3) Accreditation: studies on BME accreditation or KIMEE, an institution for the accreditation of medical education evaluation.

Table 1.

Analysis of literature related to evaluation in medical education by year

The 143 papers extracted were classified into 67 papers related to student evaluation (46.9%), 62 related to program evaluation (43.4%), and 14 related to accreditation (9.8%). According to the distribution of journals in which the 143 extracted papers were published, 73 papers (51.0%) were published in the Korean Journal of Medical Education (KJME), followed by 28 papers (19.6%) in the Korean Medical Education Review (KMER) (Table 2).

Table 2.

Analysis of literature related to evaluation in medical education by published journal

Literature related to student evaluation

Among the 143 papers, 67 (46.9%) were related to student evaluation. Among them, publications from 2001 to 2005 were the most common, followed by those from 2006 to 2010 (Table 1). The 67 student evaluation-related papers were classified into seven categories according to research content: knowledge evaluation, skill evaluation, attitude evaluation, development of teaching methods and evaluation tools, information and communication technology (IT), item analysis, and other evaluations. Papers included in each category were as follows: (1) knowledge evaluation: a study on ‘interstation works’ among academic achievement, clinical comprehensive, and clinical performance evaluations; (2) skill evaluation: research on clinical performance evaluation and clinical practice; (3) attitude evaluation: studies on relationships between patients and doctors in medical professionalism, self-directed learning, and clinical performance evaluation; (4) development of teaching methods and evaluation tools: research on standards, scales, and new teaching methods; (5) IT: research on computer-based tests, e-portfolio evaluation, and video evaluation; (6) item analysis: studies on item quality, psychometric analysis, and ability parameters; and (7) other evaluations: papers that did not correspond to the previously classified categories, such as how to use peer evaluation in problem-based learning (PBL). Most papers analyzed the validity of the evaluation tool rather than the evaluation content. As a result of the classification, skill evaluation was the most common, with 18 papers (26.9%), followed by other evaluations, with 12 papers (17.9%) (Table 3). According to the distribution of the 67 papers by journal, there were 38 papers (56.7%) in KJME, followed by 10 papers (14.9%) in KMER (Table 4).

Table 3.

Literature analysis related to student evaluation by year

Table 4.

Analysis of literature related to student evaluation by published journal

Literature related to program evaluation

Among the 143 total papers, 62 (43.4%) were related to program evaluation, and among them, publications from 2006 to 2010 (n=22) were the most common, followed by those from 2001 to 2005 (n=20) (Table 1). The research team reviewed 61 program evaluation-related studies and reclassified them into the following nine categories: (1) evaluation for curriculum; (2) evaluation for clinical practice education (evaluation for clinical practice); (3) evaluation for medical humanity education (evaluation for medical humanities); (4) evaluation for other single education subjects/courses excluding clinical practice and medical humanities education programs (evaluation for other single education subjects/courses); (5) clinical performance assessment-applied evaluation (evaluation for clinical performance assessment); (6) PBL-applied program evaluation (evaluation for PBL programs); (7) evaluation for lecture assessment; (8) indicator/conformance development and validation; and (9) others. The categories with the most papers were clinical practice program evaluation, medical humanities program evaluation, and lecture evaluation, each of which contained 10 papers (16.1% each) (Table 5). According to the distribution of the 62 papers by journal, there were 30 publications (48.4%) in KJME, followed by 10 publications (16.1%) in KMER (Table 6).

Table 5.

Literature analysis related to program evaluation by year

Table 6.

Analysis of literature related to program evaluation by published journal

Literature related to accreditation

Fourteen out of 143 papers (9.8%) were related to accreditation. The literature related to accreditation was reviewed and reclassified into the following three categories: (1) utilization and development direction of accreditation; (2) accreditation of medical education by KIMEE; and (3) development/validation of accreditation standards. As a result, the category with the highest number of accreditation-related publications was use and development direction of accreditation at nine (64.3%), and there were three publications (21.4%) in development/validation of accreditation standards (Table 7). According to the distribution of accreditation-related papers published by the journal, KMER accounted for the majority, with eight (57.1%), followed by five (35.7%) in KJME and one (7.1%) in the Journal of the Korean Medical Association (Table 8).

Table 7.

Analysis of literature related to evaluation and certification by year

Table 8.

Analysis of literature related to evaluation certification by published journal

Conclusion

This study aimed to identify research trends in the evaluation of Korean medical education. For this purpose, 143 papers were found by searching for publications related to evaluation in the medical field among papers published in domestic certified academic journals from 1995 to 2020. These papers were analyzed in relation to the publication year, journal, and research topic.

The results related to the publication year of the study were as follows. Among the 143 searched papers, 95% were published after 2000, and 62% were published during the first decade of the 21st century (2001–2010). In other words, interest in evaluation-related research in Korean medical education emerged at the beginning of the 21st century. As a result of dividing the papers into three areas (student evaluation, program evaluation, and accreditation) according to the evaluation target, there were few published papers in the 20th century, but there were relatively many papers on accreditation. In 2000, as we entered the 21st century, research on student and program evaluations increased explosively in the first 10 years. While the number of papers related to program evaluations has continued to increase in the past 5 years, the number of studies related to student evaluations has decreased.

The results related to journals were as follows. Most of the 143 papers (83.2%) were published in medical journals, including KJME (51.0%) and KMER (19.6%). However, recently, the number of papers published in KJME has gradually decreased, and no papers have been published in KJME in the last 5 years. Conversely, the number of papers published in KMER in the last 5 years has increased, and most evaluation-related papers in medical journals in the last 5 years have been published in medical journals. There were no papers published in nonmedical journals before 2000; however, in the 21st century, the number of papers published in nonmedical journals has gradually increased, and in the last 5 years, the number of papers similar to those published in medical journals has been increasing. Thus, it can be seen that researchers wanting to publish evaluation-related papers are more interested in nonmedical journals than in medical journals.

The results related to research topic analysis were as follows. Papers related to student evaluation were classified into seven categories according to their content: knowledge evaluation, skill evaluation, attitude evaluation, teaching method and evaluation tool development, IT, item analysis, and other evaluations. As a result, papers on skill evaluation accounted for the greatest number at 26.9%, and there were no other research topics that were particularly interested in research. The skill evaluation category includes research related to clinical performance and clinical practice evaluations. Student-evaluation–related papers were published the most frequently in KJME, followed by KMER.

However, looking at the trend over the past 5 years, the number of published papers related to student evaluations has sharply decreased. In addition, while no papers were published in KJME, most were published in KMER and nonmedical journals. Papers related to program evaluation were classified into the following nine categories: (1) evaluation for curriculum; (2) evaluation for clinical practice education (evaluation for clinical practice); (3) evaluation for medical humanity education (evaluation for medical humanities); (4) evaluation for other single education subjects/courses excluding clinical practice and medical humanities education programs (evaluation for other single education subjects/courses); (5) clinical performance assessment-applied evaluation (evaluation for clinical performance assessment); (6) PBL-applied program evaluation (evaluation for PBL programs); (7) evaluation for lecture assessment; (8) indicator/conformance development and validation; and (9) others. As a result, it was found that most papers were published in evaluation for clinical performance assessment, evaluation for medical humanities, lecture evaluation, and program evaluation; program evaluation was conducted in various other fields. The distribution by journal of the 62 publications related to program evaluation was found to be the highest for KJME, followed by KMER. However, over the past 5 years, the number of papers related to program evaluation has sharply declined, with no published papers in KJME and two papers published in KMER; papers published in nonmedical journals accounted for the majority during this time period.

The 14 papers related to accreditation were divided into three periods: before 2000, 2001 to 2005, and 2016 to 2020. Before 2000, all of these publications focused on the use of accreditation and direction of development. From 2001 to 2005, all papers were related to KIMEE medical education accreditation. From 2016 to 2020, all published papers were related to the use of accreditation and direction of development. Looking at the 14 papers related to accreditation by published journals, it can be seen that KMER published more papers than KJME. In particular, papers published before 2000 and between 2016 and 2020 were published in KMER, and all papers published between 2001 and 2010 were published in KJME.

Supplementary materials

Article information Continued