AGREE II Extension for Guidelines in Surgery
The AGREE II Instrument
The Appraisal of Guidelines for REsearch & Evaluation (AGREE) Instrument is a tool developed to assess the quality of guidelines and minimize the issue of variability in guideline quality. The first version was developed in 2003 and an updated version AGREE II was announced in 2017. The AGREE Collaboration defined quality of guidelines as the confidence that the potential biases of guideline development have been addressed adequately and that the recommendations are both internally and externally valid, and are feasible for practice.
The purpose of the AGREE II, is to provide a framework to:
- Assess the quality of guidelines;
- Provide a methodological strategy for the development of guidelines;
- Inform what information and how information ought to be reported in guidelines.
AGREE II instrument consists of 23 key items that can be rated in the range 1-7 (1- strongly disagree, 7- strongly agree). Items are organized into 6 domains followed by overall assessment (rating the overall quality of guideline and its recommendation for use in practice):
Domain 1. Scope and Purpose
Domain 2. Stakeholder Involvement
Domain 3. Rigour of Development
Domain 4. Clarity of Presentation
Domain 5. Applicability
Domain 6. Editorial Independence
AGREE II is applicable for the assessment of the guideline quality by healthcare workers and policy makers but also for the development of the guidelines with high methodological and reporting quality by guideline developers and educators.
Guideline Assessment Project (GAP)
The development of the AGREE II Extension for Surgical Guidelines is organised as a three staged GAP (Guideline Assessment Project) led by international multidisciplinary and interdisciplinary collaborative research working group. The aim of GAP is to develop an evidence-based and consensus-informed AGREE II Extension for Surgical Guidelines.
GAP I aimed to assess the quality of clinical practice guidelines in Surgery and to identify factors associated with quality.
The study was organised as a review of the clinical guidelines published from January 2008 to August 2017 at MEDLINE by major national and surgical organizations with an international scope.
The association between the number of guidelines published within the study period by a scientific organization, the presence of a guidelines committee, applying the GRADE methodology, consensus project design, and the presence of intersociety collaboration and guideline quality, assessed by AGREE II instrument, was investigated by two independent reviewers.
Ten surgical scientific organizations developed 67 guidelines over the study period. The median overall score using AGREE II tool was 4 out of a maximum of 7, whereas 27 (40%) guidelines were not considered suitable for use.
Guidelines produced by a scientific organization with an output of ≥9 guidelines over the study period [odds ratio (OR) 3.79, 95% confidence interval (CI), 1.01–12.66, P = 0.048], the presence of a guidelines committee (OR 4.15, 95% CI, 1.47–11.77, P = 0.007), and applying the GRADE methodology (OR 8.17, 95% CI, 2.54–26.29, P < 0.0001) were associated with higher odds of being recommended for use.
Link to the publication
The second part focused on statistical calibration of the AGREE II instrument. We employed a series of statistical methods to explore reliability and internal consistency of the AGREE II instrument. We have finally drafted a modified AGREE II document on the basis of the outcomes of statistical models.
Statistical modelling showed that excluding five items from the original tool (items 1, 2, 5, 7 and 8) and rearranging the remaining items into four domains instead of six would enhance the instrument. We have finally drafted a modified AGREE II document for guidelines in surgery, on the basis of the outcomes of statistical models.
The third part of the project aims to use the information from the previous GAP projects to develop the extension document using a structured Delphi process involving relevant stakeholders.
The executive group consists of surgeons, members of surgical quality and research boards, guideline developers, evidence synthesis experts, GRADE methodologists, biostatisticians, and 2 leads of the AGREE Group, divided into four working groups which have discussed the findings of previous work, defined the methodology and study design, and identified potential stakeholder groups to comprise the Delphi panel.
Following the Delphi process, the executive group will meet to discuss the findings and compose the first draft of the extension document, conduct its pilot testing and compose AGREE II extension statement and extension checklist.
The executive group will monitor the use of the extension document and appraise its applicability in surgical guidelines for a reasonable period of time after dissemination and will publish their findings. Following consideration of the outcomes, feedback, criticism, suggestions and new evidence in the field, the executive group will discuss the need for an update.