Interobserver agreement among clinicians in the treatment recommendations for patients with unruptured intracranial aneurysms

Article information

Korean J Cerebrovasc Surg. 2025;.jcen.2025.E2025.07.001

Publication date (electronic) : 2025 October 31

doi : https://doi.org/10.7461/jcen.2025.E2025.07.001

O-Ki Kwon^1,2, Seung Pil Ban^,1,2, Young Deok Kim¹, Hwan Seok Shim¹, Seung Bin Sung¹

¹Departments of Neurosurgery, Seoul National University Bundang Hospital, Seongnam-si, Gyeonggi-do, Korea

²Departments of Neurosurgery, Seoul National University College of Medicine, Seoul, Korea

Correspondence to Seung Pil Ban Department of Neurosurgery, Seoul National University Bundang Hospital, 82 Gumi-ro 173 beon-gil, Bundang-gu Seongnam-si, Gyeonggi-do, 13620, Korea Tel +82-31-787-7175 Fax +82-31-787-4097 E-mail neurosurgeryban@gmail.com

Received 2025 June 30; Revised 2025 October 3; Accepted 2025 October 18.

Abstract

INTRODUCTION

Do clinicians need to provide the same recommendation for a given patient with unruptured intracranial aneurysms (UIAs)? Thanks to evidence from various studies, we now have general estimates of the risks associated with aneurysm characteristics such as size, location, contour, and many other factors [5,13]. However, evidence is still insufficient to predict precise future bleeding risks. Even if such predictions were possible, actual outcomes could be influenced by many other factors, such as the clinician’s ability, experience, philosophy, and the supporting systems including medical staff and equipment. Personal, regional and national socioeconomic factors, such as the health care and medical insurance systems, could also influence the clinician’s decision. Therefore, variability in management decisions for UIAs among clinicians, institutions, regions, and countries may exist and is understandable [3]. However, would it also be acceptable among clinicians working at the same institution and specialty? If a patient is offered different management plans depending on which clinician they happen to see, this would not be a desirable situation from the patient’s standpoint.

This study was conducted to assess interobserver agreement on clinicians’ recommendations for patients with UIAs among three endovascular neurosurgeons working at a single center. The grade of recommendation (GOR) system consisting of six grades, which we developed and have used in practice, was used as a tool to identify and compare clinicians’ decisions.

MATERIALS AND METHODS

The institutional review board approved this study, and informed consent was waived. All the three observers (SPB, YDK, and OKK), each with more than five years of experience in endovascular aneurysm therapy, worked at the same institution as neurointerventionists and shared the same background. Patients with newly diagnosed UIAs who visited a senior doctor (observer A) at the outpatient clinic during the first five months of 2015 were reviewed. The year 2015 (7 years prior to the study) was arbitrarily selected so that this study would not affect the care of actual patients. In 2015, observers B and C were not yet working as faculty members at the center. The GOR system was first introduced in practice in 2020; therefore, GORs were not documented in the medical records of the study cases. The total number of patients was 272. Among them, 202 aneurysms from 161 patients, for which aneurysm characteristics could be reviewed by digital subtraction angiography (DSA), were included. The data provided to observers consisted of images of the first DSA and the initial outpatient clinic medical records, including age, gender, race, symptoms, past history, aneurysm growth tendency, and other information relevant to determining the indication for treatment. To prevent influence from previous clinical decisions, other medical records and imaging studies reflecting actual management were not provided. After review, the observers assigned a GOR for each aneurysm. The observers applied the GOR system independently based on their current knowledge, experience, and clinical philosophy.

In this article, we use “treatment” to indicate aggressive treatments such as endovascular or surgical repair of UIAs and “no-treatment” for conservative management including observation, follow-up, medication, and risk factor control. The “management” indicates all management including treatment and no-treatment.

GOR system

The GOR system consists of 6 grades. It was designed to convey clinicians’ recommendations to patients more clearly, enabling patients and their family members to understand the clinician’s general intent and the rationale behind the recommendations. It has also been used to facilitate clear communication among medical staff. Grades are classified based on three components: (1) the clinician’s opinion regarding treatment; (2) the clinician’s assessment of aneurysm risks; and (3) the clinician’s planned approach if the patient chooses not to follow the recommendation. The system serves as a language framework for expressing clinicians’ advice and recommendations to individual patients. The authors have applied this system in clinical practice. The GOR system is briefly summarized in Table 1.

Table 1.

Summary of the grade of recommendations

Grade 1.

- Treatment is recommended.

- The strength of this recommendation is strong.

- Your doctor believes that the anticipated risks of the aneurysm are high, considering the aneurysm characteristics, your age, and overall health status.

- If you (patient) strongly prefer not to undergo treatment, your doctor will not accept your decision. Your doctor believes that treatment is necessary.

- In the case of no-treatment, treatment will be advised again during outpatient department (OPD) follow-up unless there are significant changes in the anticipated risks. If significant changes occur, the grade may be adjusted.

Grade 2.

- Treatment is recommended.

- The strength of this recommendation is moderate.

- Your doctor believes that the aneurysm carries some risk, but the anticipated risks are not very high, considering the aneurysm characteristics, your age, and overall health status.

- If you (patient) strongly prefer not to undergo treatment, your doctor may respect your decision and reassure you that the aneurysm risk is not very high, although the doctor still believes that treatment would be advisable.

- In the case of no-treatment, treatment may be advised again during OPD follow-up. If there are significant changes in risk, the grade may be adjusted.

Grade 3.

- Treatment and no-treatment are equally recommended.

- The strength of this recommendation is modest.

- Your doctor believes that the anticipated risks of the aneurysm are not high. The risks are modest, considering the aneurysm characteristics, your age, and overall health status.

- Your doctor will ask for your opinion regarding treatment or no-treatment.

- Even if you (patient) do not want to make a choice or prefer not to be involved in the decision, your doctor will still seek your opinion. If necessary, your doctor will provide additional information and more time to consider your options.

- When you choose either treatment or no-treatment, your doctor will ask for the reasons behind your choice to ensure that your decision is based on accurate scientific information. If not, your doctor may provide clarification and allow you additional time to reconsider.

- Your doctor will respect your decision.

- In the case of no-treatment, treatment may not be recommended again during OPD follow-up unless there are significant changes in the anticipated risks. If significant changes occur, the grade may be adjusted.

Grade 4.

- No-treatment is recommended.

- The strength of this recommendation is moderate.

- Your doctor believes that the aneurysm carries some risk, but the anticipated risk is low, considering the aneurysm characteristics, your age, and overall health status.

- If you (patient) strongly prefer to undergo treatment, your doctor may respect your decision. However, your doctor still believes that treatment is not necessary.

- In the case of no-treatment, treatment may not be advised again during OPD follow-up unless there are significant changes in the anticipated risks. If significant changes occur, the grade may be adjusted.

Grade 5.

- No-treatment is recommended.

- The strength of this recommendation is strong.

- Your doctor believes that the aneurysm carries some risk, but the anticipated risk is very low, considering the aneurysm characteristics, your age, and overall health status.

- If you (patient) strongly prefer to undergo treatment, your doctor will not accept your decision. Your doctor believes that treatment is not necessary and sometimes not appropriate in medical ethics.

- In the case of no-treatment, treatment may not be advised again during OPD follow-up unless there are significant changes in the anticipated risks. If significant changes occur, the grade may be adjusted.

Grade 6.

- Your doctor believes that the anticipated aneurysm risk is high; therefore, treatment should be recommended, as in Grade 1 or 2. However, for you, additional important factor(s) must be considered.

- These factors may include:

1) excessively high anticipated treatment risk, and

2) minimal expected benefit from treatment due to a limited life expectancy associated with advanced age or coexisting serious diseases/conditions.

- Your doctor will engage in discussions with you to identify the most appropriate management options.

- In the case of no-treatment, your doctor may ask your opinions again during OPD follow-up. If significant changes occur, the grade may be adjusted.

RESULTS

The characteristics of the cases are shown in Table 2. A total of 202 aneurysms were identified in 161 patients. Of these, 118 (73%) were female. The mean age was 57.7±10.6 years (range, 26-83). Multiple aneurysms (2-4 aneurysms) were observed in 37 patients. Six aneurysms were fusiform, two were possible vertebral artery dissections, and the remainder were saccular aneurysms. Irregular aneurysm morphologies such as daughter sacs, eccentric parts, lobulations, and elongations were present in 100 aneurysms. A history of minor stroke was found in 7 patients, subarachnoid hemorrhage from other aneurysms (not included in the present study) in 6, intracerebral hemorrhage in 2, and coronary artery stenting in 1. At the time of the first OPD visit, none of the patients had neurological deficits that interfere with normal daily life. The patients had no presenting symptoms, or if any, nonspecific ones such as mild headache, dizziness, or syncope. No specific genetic or familial background was identified in these cases. The grading assigned by individual observers are shown in Table 3. The average proportion of treatment recommendations (GOR 1 and 2) was 59%, while no-treatment recommendations accounted for 24%. GOR 3 was selected in 18% of cases. Only one observer assigned GOR 6 for a single aneurysm. The overall Fleiss’ kappa value was 0.52, indicating a moderate level of interobserver agreement among the three observers. Agreement was relatively high for GOR 1 and GOR 5, and lowest for GOR 3. The kappa values among the three observers and between two observers are shown in Table 4 and Fig. 1.

Table 2.

Summary of the characteristics of cases

Table 3.

The grade of recommendations marked by three observers

Table 4.

The kappa values of interobserver agreement among the three observers

Fig. 1.

The kappa values of interobserver agreement among the three observers. A, B and C indicates each observer. GOR, grade of recommendation

When the grades were classified into the treatment recommendation (Tx RC) group (GOR 1 and 2), middle group (GOR 3), and no-treatment recommendation (No-Tx-RC) group (GOR 4 and 5), the overall kappa value was 0.84, indicating an almost perfect level of interobserver agreement. The level of interobserver agreement was high in both the Tx RC group and No-Tx-RC group. It was the lowest in GOR 3 (Table 5 and Fig. 2).

Table 5.

The kappa values according to the simpler classification

Fig. 2.

The kappa values according to the simpler classification. A, B and C indicates each observer Tx RC, treatment recommendation (GOR 1 and 2); Middle, (GOR 3); No Tx RC, no-treatment recommendation (GOR 4 and 5). GOR, grade of recommendation

DISCUSSION

Decision-making in clinical practice should be based on evidence. With respect to the risks of UIAs, general estimates are now available regarding outcomes associated with aneurysm characteristics such as size, location, contour, and other factors [2,12,14,16,17,20,21]. However, these data are often insufficient to predict precise future outcomes. Previous studies have also shown that clinicians may disagree on the management of UIAs, even when they share a background in the same specialty, similar capabilities in aneurysm management, or years of practice [3]. Defining each characteristic may be complex and confusing [7,15]. Moreover, clinicians do not make decisions solely based on anatomical aneurysm risks. Numerous additional factors are considered, including patient-specific characteristics such as age, medical conditions, psychological traits (anxiety and fear), and family attitudes. Socioeconomic factors, including the medical insurance system, may also play a role. Both personal and institutional experience with treatment outcomes can influence decision-making. Priorities may vary among clinicians, institutions, and countries. Interpretation and application of existing evidence to individual patients can also differ.

This study demonstrated both similarities and differences in recommendations. With respect to treatment versus no-treatment, a high level of agreement was observed. It is understandable that identifying high-risk aneurysms is relatively straightforward for neurointerventional specialists, as evidence regarding risks associated with aneurysm characteristics is well established. However, agreement at a more detailed level was only moderate, indicating that specific recommendations varied among clinicians. Differences between adjacent grades in the GOR system reflect the strength of clinicians’ recommendations. Compared with GOR 1, GOR 2 allows greater flexibility for patients to consider their own options. A similar distinction exists between GOR 4 and GOR 5. Differences between GOR 2 and GOR 3, as well as between GOR 3 and GOR 4, are also substantial. Ultimately, the final decision is made by the patients; however, patients generally follow their doctor’s recommendations, trusting that the clinician will select the best options. Variability in clinicians’ recommendations may lead to patient confusion and concern.

Our study shows the lowest level of interobserver agreement in GOR 3. GOR 3 grading represents a small aneurysm with additional factors such as location or morphology that confer a moderate level of risk. The findings indicate that observers may differ in their assessment of whether treatment is warranted for these aneurysms. This result is consistent with a previous study [3], which showed that, even among physicians sharing a background in the same specialty, decisions regarding the management of UIAs may vary individually. Although the present study demonstrated low interobserver agreement in GOR 3 grading, we consider GOR 3 grade to be an essential component of the overall GOR grading system. However, for patients with UIAs classified as GOR grade 3, there may be concerns that such categorization could actually create confusion in treatment decision-making. Nevertheless, as mentioned above, in the treatment of UIAs, it is inevitable that there are situations where treatment decisions cannot be made definitively due to multiple factors, and such decisions may vary depending on the physician. Therefore, when an aneurysm is determined to fall into GOR grade 3, physicians can explain the various factors influencing treatment decisions to patients and inform them that opinions may differ among physicians. Likewise, when physicians encounter an aneurysm that has been classified as GOR grade 3 at another institution, they may recognize that the decision regarding treatment is inherently equivocal. This classification, therefore, serves as a framework to support comprehensive patient counseling and to facilitate shared decision-making in determining the optimal management strategy.

Do clinicians need to provide the same recommendation for the same patient with UIAs? Variability in clinical decision-making across clinicians, institutions, regions, and countries can exist and should be acknowledged. Nevertheless, consistency in recommendations is desirable, at least among physicians working at the same hospital with similar backgrounds. In our study, we believed that a high level of consensus existed regarding the management policy for patients with UIAs. This was observed in the context of treatment versus no-treatment. However, a lower level of agreement was noted when examining the details of specific recommendations. The observers are working at the same hospital with almost same backgrounds. Patient backgrounds were also similar, and they were all Koreans. There is no big problem or disparity in the patients’ socioeconomic environments. The incidence of cerebral aneurysms is high in South Korea [10,11]. Coverage of UIA treatment by the national health insurance enables patients to seek consultations at multiple hospitals and make informed decisions regarding treatment at a relatively lower cost. In contrast, in other countries, the cost of intracranial aneurysm treatment is much higher than in Korea, amounting to $35,446 in the United States and €14,477 in France [1,8]. As a tertiary general hospital in South Korea, our institution has both the neuroendovascular center and the vascular neurosurgery section in department of neurosurgery. In the neuroendovascular center, about 1,000 patients a year are treated by endovascular means, including 600-700 endovascular treatments for aneurysms. About 1,500 newly diagnosed intracranial aneurysm patients also visit the outpatient clinic. The overall complication rate of UIA coiling during the recent 5 years was about 0.8% per year. It was almost the same among the three observers. The observers B and C were trained as clinical fellows by observer A. In terms of experience, observer A has about 3,600 aneurysm coiling cases, observer B has about 1,200 and C has about 300 cases. The level of experience did not affect the agreement level. The kappa value between A and C was higher than between A and B, in this study.

Although approaches such as the PHASES score quantitatively assess rupture risk by incorporating multiple clinical factors [5], the present study on GOR grading adopts a different perspective. Specifically, it stratifies the decision regarding UIA management into six categories based on expert knowledge and clinical experience in assessing aneurysm risk. This grading system provides a structured and accessible framework for communicating treatment considerations to both physicians and patients, thereby facilitating clearer clinical guidance. In practice, this study was very helpful for us. After the study, it became easier to reach a consensus that we need to revise our scientific meetings and conferences. Each staff member had a chance to review and reconsider their decision process. We realized the presence of discrepancy among us. Of course, we still believe that some differences should be allowed as a clinician’s personal discretion. The reasons for disagreement and agreement need to be further identified and analyzed. The agreement level at our hospital may not be of interest to other professionals. However, if we can see similarities and differences among clinicians at different hospitals in different regions and countries, it would give us implications about what and why they value various factors and how they interpret risks and benefits. Additionally, for patients, the necessity of treating UIAs can be explained in a stratified manner, making it easier for them to understand. This approach may enable patients to make more informed decisions regarding treatment and help alleviate some of the vague fears associated with having UIAs.

This study may have several limitations. First, the analysis was based on retrospective patient data; however, the primary objective was to assess the level of agreement among clinicians, rather than to evaluate prospective outcomes or the impact of clinicians’ decisions on patient outcomes. Therefore, prospective data collection was not essential. Second, observers may have assigned GOR grading not strictly according to their actual clinical policies, but in ways that they perceived as socially or professionally acceptable to other specialists, a phenomenon known as desirability bias [4,18,19]. Although this may obscure some aspects of true clinical behavior, a clinician’s desirability still reflects their genuine beliefs. This study did not analyze how often treatment was recommended or withheld, nor did it assess the GOR grading according to aneurysm characteristics. Third, specific patient factors-such as severe comorbidities, psychological issues, extreme age, or unusual socioeconomic circumstances-were not considered. The GOR system cannot capture all nuances of clinicians’ intentions or recommendations; no system can fully encompass the subtleties of clinical judgment. In practice, additional explanations for individual patients are often provided, and in our experience, these can be integrated within the GOR framework. Finally, this study did not analyze the principles guiding the choice between surgical and endovascular treatments. The GOR system focuses solely on treatment recommendations, independent of treatment modality. While treatment options are always considered together in practice, the decision-making processes for surgical versus endovascular interventions differ. We consider the selection of treatment modality to be a subsequent step following the fundamental decision of whether to treat or not.

CONCLUSIONS

This study demonstrates a high level of interobserver agreement regarding treatment versus no-treatment for UIAs among experts with the same background. However, some differences were observed in detailed treatment recommendations. Further studies are needed to clarify the specific reasons for these similarities and differences. In addition, this study suggests that the GOR system can serve as a tool to evaluate agreement levels among clinicians.

Article information Continued