Critical Appraisal of the National Institute for Health and Care Excellence (NICE) Guidelines for Spine Disorders using the Appraisal of Guidelines for Research and Evaluation II Instrument (AGREE II)

Ning Liang^1,2*, Sizhan Wu², Simon Roberts³, Navnit Makaram⁴, James Reeves Mbori Ngwayi¹ and Daniel Porter^1,2
*Correspondence: Ning Liang, Department of Clinical Medicine, Tsinghua University, China, Email:

Keywords

AGREE II; Clinical Practice Guideline; NICE; Spine

Background

The World Health Organization reported in 2021 that musculoskeletal diseases are the main cause of global disability with approximately 1.71 billion people affected worldwide [1]. In 2019 the prevalence of low back pain was estimated as 568 million and neck pain 223 million [2]. An aging population will result in increasing numbers of people with these and other common spine disorders [3].

Clinical practice guidelines can provide health care providers with decision making recommendations from an evidence base [4]. Around the world, many institutions, organisations or groups have formulated and issued clinical practice guidelines. The National Institute for Health and Care Excellence (NICE), was established as a special health authority in 1999 and a non-departmental public body in 2013, performing statutory functions in the United Kingdom with the aim of ‘improving health and wellbeing by putting science and evidence at the heart of health and care decision making [5]. To date, more than 1750 different guidelines have been formulated under several headings, with spine disorders contributing significantly to the burden of disease, NICE guidance in this clinical area encompasses several guideline types, including: Clinical Guidelines (abbreviation CG) which were succeeded by NICE guidelines (NG) in 2015 these guidelines review the evidence across broad health care topics; Interventional Procedure Guidance (IPG) which review the efficacy and safety of procedures; Technology Appraisal Guidance (TG) which review clinical and cost effectiveness of new treatments; Medical Technology Guidance (MTG) which review new medical technologies for adoption in the UK National Health Service for multiple clinical conditions [5]. The effectiveness of a clinical practice guideline is dependent on its inherent quality. During guideline development the World Health Organization (WHO) recommended that “Prior to submission for clearance, the AGREEII appraisal instrument should be used to check whether the guideline meets international quality standards and reporting criteria” [6]. In 2010 AGREE II was developed and formulated by an international development and research team, based on quantitative scores to evaluate the quality of the guideline [7]. Since then NICE has undertaken internal audit to ensure that the processes and methods for guideline development are based on internationally accepted criteria of quality, as detailed in the AGREE II instrument [8]. However internal quality assurance processes may not reflect the evaluation of external auditors. Furthermore, many guidelines were developed prior to the adoption of these standards use this style when you need to begin a new paragraph.

The AGREE II tool has been used independently to evaluate several NICE guidelines, involving urological and endocrine system disorders [9-12], but there are currently no studies using the AGREE II tool to evaluate NICE guidelines for musculoskeletal or neurosurgical conditions. The purpose of this study is to use the AGREE II tool to assess quality of NICE guidelines for spine disorders.

Material and Methods

Search keyword methodology

Our goal was to identify guidelines related to spine disorders, defined as disorders of the musculoskeletal structures surrounding the spinal neural elements. Two researchers independently performed keyword searches for spine disorders on the official website of International Classification of Diseases 11^th Revision (ICD-11) (https://icd.who.int/browse11/l-m/ en). Search terms identified via dictionary linkage are shown in Electronic Supplementary Material S1. Search keywords retrieval process followed the PRISMA flow algorithm [13]. Inclusion and exclusion criteria for categories of disorders searched in the official website of ICD-11 are shown in Electronic Supplementary Material S2 and S3 respectively. In our study, a total of 28 search keywords for spine disorders were formed from ICD-11 and are shown in Electronic Supplementary Material S4. Use this for the first paragraph in a section, or to continue after an extract.

Guideline identification

The search for spine guidelines was carried out by two researchers (first author and second author) independently using the NICE website (https:// www.nice.org.uk/) up to 31 December 2019. The guideline search used keywords obtained in the aforementioned process. Both keyword and manual searches were performed.

Inclusion criteria were:

1. Literature related to spine disorders

2. Literature meets the guidelines standard of the National Guidelines Clearinghouse [14].

Exclusion criteria were:

1. Disorders secondary or metastatic without causing spinal cord compression

2. Systemic diseases including sites other than the spine

3. Physiological or pathological abnormalities of spinal cord, spinal neural structures, and vertebral vascular conditions caused by disorders not primarily of the spine. The guidelines retrieval process followed the PRISMA flow algorithm [13].

AGREE II instrument

The AGREE II assessment system is an internationally validated tool for assessing guideline quality, including 23 main items in 6 domains and 2 overall assessment items (Table 1). Each domain addresses an aspect of guideline quality, namely: "scope and purpose", "stakeholder involvement", "rigour of development", "clarity of presentation", "applicability", and "editorial independence". The 23 field items and one of the overall assessment items are graded on a seven point Likert scale. Item scores range from 1 (no information or very poor quality) to 7 (all conditions met and of excellent quality) [7].

Ethics statement

This study is an evaluation of existing literature without human subjects; hence it is not subject to ethics committee evaluation.

Evaluation of the guidelines

Each guideline was assessed by a panel of four appraisers. All appraisers were familiar with the AGREE II instrument having used it before to evaluate clinical guidelines and completed the online training tools recommended by AGREE II (www.agreetrust.org) [7]. No communication between appraisers occurred during the rating process. Data analysis was performed after completing the evaluation of all NICE spine related guidelines.

Statistical analysis

The score for each domain was obtained by the sum of all scores of the individual items in a domain and then standardized as follows: (obtained score minimum possible score)/(maximum possible score minimum possible score) [7]. Mean values and 95% confidence intervals (CI) for all raters were calculated. Although domain scores can be used to compare different guidelines and to help determine whether guidelines should be recommended, the AGREE II tool does not set a minimum domain score, nor does it define the boundary criteria for identifying the quality of the guidelines. These decisions are made by the user. According to convention in existing research reports, the domain score criteria we used were: <40% very low quality, 40%~59% low quality, 60%~79% acceptable quality, ≥ 80% good quality [15,16].

Scores obtained for individual items within domains were calculated using the same method as for domain scores. Since the purpose of the AGREE scale is to emphasise and encourage best practice, we took the view that domains which failed to reach ‘acceptable’ quality threshold should not be the focus of detailed critique. Consequently, although all domain scores are presented, individual item scores are only displayed and discussed for domains which reach at least ’acceptable’ quality.

For overall guideline assessment, appraiser scores for item 1 of the ‘overall assessment’ section were used to derive a score for each guideline by the same method as for domain scores.

Statistical analysis of the data was performed using Statistical Package for the Social Science (SPSS Inc, Chicago, Illinois, USA) version 22.0 and Stata (Version 16.1, StataCorp LP, College Station, TX) software programs. Correlation between overall score and guideline publication date was calculated using Spearman’s test. Inter rater reliability of domain scores was assessed using intra-class correlation coefficient (ICC). ICC values less than 0.40, between 0.40 and 0.59, between 0.60 and 0.74, and greater than 0.75 were indicative of poor, moderate, good, and excellent reliability, respectively [17]. Mann-Whitney U test was used to investigate the quality differences between guideline types. The level of statistical significance was set at p<0.05.

Results

A total of 37 guidelines fulfilled the inclusion criteria, including 29 IPG, 1 MTG, 4 NG or CG and 3 TA (Table 1). The scores of each domain and overall scores in the 37 guidelines after evaluation by AGREE II criteria are shown in Table 2. All four CG/NG guidelines were categorized as having ‘acceptable’ quality in all domains. Seven (7/29) IPG guidelines and one (1/3) TA guideline were categorized as having ‘acceptable’ quality in all domains. The overall scores in 26 NICE spine related guidelines were above the “acceptable” level. The mean overall score of all NICE spine related guidelines was 68.8% (95% CI: 62.3%~75.3%).

Mean domain and item scores are shown in Table 3. Mean domain score is highest for ‘Clarity of Presentation’ (73.6%), followed by ‘Scope and Purpose’ (73.2%), ‘Rigour of Development’ (68.1%), ‘Editorial Independence’ (64.5%) and ‘Stakeholder Involvement’ (63.9%). Mean domain score for ‘Applicability’ (53.2%) is the lowest.

Five domains exceeded the threshold for ‘acceptable’ quality (mean score of 60%): ‘Scope and Purpose’, ‘Stakeholder involvement’, ‘Rigour of development’, ‘Clarity of presentation’, and ‘Editorial Independence’. The quality evaluation of two items in these five domains fell below this 60% threshold for acceptability; item 5 ‘the views and preferences of the target population (patients, public, etc.) have been sought’ and item 13 ‘the guideline has been externally reviewed by experts prior to its publication’ (Table 3) (Figure 1).

Mean domain score for IPG and non-IPG documents are 64.1 (95% CI: 58.0~70.1) and 73.4 (95% CI: 67.3~79.4) respectively (z=-2.085, p=0.037) (Table 4). Significant differences in domain scores for domains 1, 2, 4 and 5 were also found (Table 4). Non-IPGs also have higher overall scores (78.1) than IPGs (66.2) (z=-2.687, p=0.007) (Table 4) (Figure 2).

The overall scores in 26 NICE spine related guidelines were above the “acceptable” level. The mean overall score of all NICE spine related guidelines was 68.8% (95% CI: 62.3%~75.3%) (Table 2). Intraclass correlation (ICC) values for each NICE guideline ranged from 0.393 to 0.953 (Table 5).

Discussion

The purpose of NICE at its inception was ‘to create consistent guidelines and end rationing of treatment by postcode across the UK [18]. Ours is the first study to use the AGREE II instrument to assess the quality of NICE guidelines for spine related disorders. Up to end of 2019, we identified 37 guidelines which fulfilled our inclusion criteria as related to spine disorders. This represented approximately 3% of the total guideline cohort in the NICE library at that time point.

Reliability

Intraclass correlation (ICC) of overall guideline scores in our study ranged from 0.39 to 0.95. 35 out of 37 guideline evaluations (94.6%) were categorized as exhibiting good or excellent inter rater reliability.

Domains reaching acceptability threshold

The highest mean score was for the domain “clarity of presentation”, with mean scores of a further four domains (‘scope and purpose’, ‘rigour of development’, ‘editorial independence’ and ‘stakeholder involvement’) also exceeding the 60% threshold. Evidence based recommendations in NICE guidelines include that they should be ‘developed by independent committees, including professionals and lay members, and consulted on by stakeholders [19]. Since the first guideline was released in 1999, NICE has published over 1750 guidelines, and technology appraisal guidance alone exceeds five hundred in number [18]. After more than 20 years of development, NICE has accumulated considerable experience in their formulation and publication [20]. However, items 5 and 13 failed to reach acceptability threshold in any of the guidelines; finding patients willing and able to provide input into guideline development has proved difficult, for example several guidelines report that ‘NICE’s Patient and Public Involvement Programme were unable to gather patient commentary for procedures under evaluation’. NICE recognises the need to support patients, nursing staff, and the public to participate in the development and formulation of the guidelines, and has taken measures to increase the participation of these personnel, such as establishing a Public Involvement Programme and Citizens Council project [21]. High quality guidelines should be externally reviewed by experts prior to their publication; however it appeared to some assessors that there may have been a lack of engagement from key stakeholders identified as important contributors by NICE in the consultation phase of guideline development.

Domains not reaching acceptability threshold

Although mean domain ‘Applicability’ scored below 60%, more than onethird (14/37) of the guidelines scored this domain at ‘acceptable’ level or above. All NG/CGs scored over 60%. All guidelines contained accessible documents to assist doctors putting ‘guidance into practice’. However in many cases, assessors may have found these documents non guideline specific. Four previous NICE guidelines evaluated using AGREE II resulted in a wide range of scores in this domain, with some authors agreeing that applicability represented their weakest domain (scores of 48 and 56 [11- 12]), whereas others rated it highly (scores of 82 and 100 [9,10]). The applicability of a guideline is key to its success but may be dependent on heterogenous structural factors within the National Health Service systems, and therefore requires independent consideration.

Overall guideline assessment

Regarding overall evaluation, the AGREE II manual does not describe how to perform quantitative scoring [22]. Previous studies have applied domain score calculation methods to calculate mean scores for the item ‘rate the overall quality of this guideline’ without reference to the item ‘recommendations of the guideline for use’ [23,24]. In others, assessors have scored this based on the average rating given to the six domains [9,25]. In our study we gave no instructions to reviewers about providing overall recommendations. The mean overall score for all NICE spine disorder guidelines exceeded the 60% threshold for acceptability and a majority of assessors recommended every guideline for use.

Overall guideline scores were significantly lower in IPGs and this was observed across several domains. IPGs are of more limited scope than other guidelines, especially in particular CG/NGs in which supporting evidence can run to several thousand pages.

Overall guideline evaluation scores correlated significantly with year of publication, suggesting a dynamic process of continuous guideline quality improvement.

Limitations

Our study has several limitations. First, AGREE II remains a subjective evaluation tool. Second, the method for calculating a consensus derived overall guideline score using AGREE II is established neither by developer instructions nor by precedence in the literature. Consequently, the method we chose may be considered arbitrary (although consistent with the calculation method of AGREE II domain and item scores). Although inter rater reliability was good or excellent in 95% of evaluations, the lack of agreement between assessors in a small number may weaken reliability of the method. We attempted to overcome this by fulfilling training and assessor number recommendations beforehand.

Conclusion

Our consensus is that the NICE spine disorder guidelines should be recommended for clinical practice as they demonstrate either acceptable or good overall quality. Evidenced ongoing quality improvement over time continues to be reassuring.

Disclosure of Interest

The authors report there are no competing interests to declare.

Statement of Using Databases

The International Classification of Diseases 11th Revision database was publicly available during the study period. The authors confirm that all methods were carried out in accordance with relevant guidelines and regulations.

Data Availability Statement

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

Funding Details

None.

Acknowledgement

None.

References

1. World Health Organization. “Musculoskeletal conditions.” Accessed August 2021.
2. Cieza, A, Causey K and Kamenov K. “Global estimates of the need for rehabilitation based on the Global Burden of Disease study 2019: A systematic analysis for the global burden of disease study 2019.” Lancet 396(2021):2006-2017.

   [Crossref] [Google Scholar]

3. Hartvigsen, J, Hancock MJ and Kongsted A. “What low back pain is and why we need to pay attention.” Lancet 391(2018):2356-2367.

   [Crossref] [Google Scholar]

4. Woolf, SH, Grol R and Hutchinson A. “Clinical guidelines: Potential benefits, limitations, and harms of clinical guidelines.” BMJ 318(1999):527-530.

   [Crossref] [Google Scholar]

5. Musculoskeletal conditions
6. WHO handbook for guideline development, 2nd Edition. Accessed August 2021.
7. AGREE II-AGREE Enterprise website. Accessed August 2021.
8. Developing NICE guidelines: The manual. Accessed 2021 Oct.
9. Chua, ME, Mendoza J and See M. “A critical review of recent clinical practice guidelines on the diagnosis and treatment of non-neurogenic male lower urinary tract symptoms.” Can Urol Assoc J 9(2015):E463-E470.

   [Crossref] [Google Scholar]

10. Li-Zhen, L, Yun X and Xiao-Dong Z. “Evaluation of guidelines on the screening and diagnosis of gestational diabetes mellitus: Systematic review.” BMJ Open 9(2019):e023014. [Crossref]

    [Google Scholar]

11. Enikeev, D, Misrai V and Rijo E. “EAU, AUA and NICE Guidelines on surgical and minimally invasive treatment of benign prostate hyperplasia: A critical appraisal of the guidelines using the AGREE-II tool.” Urol Int 9(2021):1-10.

    [Crossref] [Google Scholar]

12. Perez-Panero, AJ, Ruiz-Munoz M and Cuesta-Vargas AI. “Prevention, assessment, diagnosis and management of diabetic foot based on clinical practice guidelines: A systematic review.” Med Balt 98(2019):e16877.

    [Crossref] [Google Scholar]

13. Moher, D, Shamseer L and Clarke M. “Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement.” Syst Rev 4(2015):1.

    [Crossref] [Google Scholar]

14. Global estimates of the need for rehabilitation based on the Global Burden of Disease study 2019: A systematic analysis for the global burden of disease study 2019
15. Doniselli, FM, Zanardo M and Manfre L. “A critical appraisal of the quality of low back pain practice guidelines using the AGREE II tool and comparison with previous evaluations: A EuroAIM initiative.” Eur Spine J 27(2018):2781-2790.

    [Crossref] [Google Scholar]

16. Sekercioglu, N, Al-Khalifah R and Ewusie JE. “A critical appraisal of chronic kidney disease mineral and bone disorders clinical practice guidelines using the AGREE II instrument.” Int Urol Nephrol 49(2017):273-284.

    [Crossref] [Google Scholar]

17. Cichetti, DV. “Guidelines, criteria and rules of thumb for evaluating normed and standardized assessment instruments in psychology.” Psychol Assessment 6(1994):284-290.

    [Crossref] [Google Scholar]

18. History of NICE. Accessed August 2021.
19. NICE-The National Institute for Health and Care Excellence. Accessed August 2021.
20. NICE: 20 years of evidence-based decision making. Accessed August 2021.
21. Patient and public involvement policy. Accessed 2021 Oct.
22. Yaman, ME, Gudeloglu A and Senturk S. “A critical appraisal of the north american spine society guidelines with the appraisal of guidelines for research and evaluation II instrument.” Spine J 15(2015):777-781.

    [Crossref] [Google Scholar]

23. Sabharwal, S, Patel NK, Gauher S. “High methodologic quality but poor applicability: Assessment of the AAOS guidelines using the AGREE II instrument.” Clin Orthop Relat Res 472(2014):1982-1988.

    [Crossref] [Google Scholar]

24. Gavriilidis, P, Roberts KJ and Askari A. “Evaluation of the current guidelines for resection of hepatocellular carcinoma using the appraisal of guidelines for research and evaluation II instrument.” J Hepatol 67(2017):991-998.

    [Crossref] [Google Scholar]

25. Hatakeyama, Y, Seto K and Amin R. “The structure of the quality of clinical practice guidelines with the items and overall assessment in AGREE II: A regression analysis.” BMC Health Serv Res 19(2019):788.

    [Crossref] [Google Scholar]