BACKGROUND AND OBJECTIVES:

Guideline recommendations for the same clinical condition may vary. The purpose of this study was to determine the degree of agreement among comparable asthma and bronchiolitis treatment recommendations from guidelines.

METHODS:

National and international guidelines were searched by using guideline databases (eg, National Guidelines Clearinghouse: December 16–17, 2014, and January 9, 2015). Guideline recommendations were categorized as (1) recommend, (2) optionally recommend, (3) abstain from recommending, (4) recommend against a treatment, and (5) not addressed by the guideline. The degree of agreement between recommendations was evaluated by using an unweighted and weighted κ score. Pairwise comparisons of the guidelines were evaluated similarly.

RESULTS:

There were 7 guidelines for asthma and 4 guidelines for bronchiolitis. For asthma, there were 166 recommendation topics, with 69 recommendation topics given in ≥2 guidelines. For bronchiolitis, there were 46 recommendation topics, with 21 recommendation topics provided in ≥2 guidelines. The overall κ for asthma was 0.03, both unweighted (95% confidence interval [CI]: −0.01 to 0.07) and weighted (95% CI: −0.01 to 0.10); for bronchiolitis, it was 0.32 unweighted (95% CI: 0.16 to 0.52) and 0.15 weighted (95% CI: −0.01 to 0.5).

CONCLUSIONS:

Less agreement was found in national and international guidelines for asthma than for bronchiolitis. Additional studies are needed to determine if differences are based on patient preferences and values and economic considerations or if other recommendation-level, guideline-level, and condition-level factors are driving these differences.

What’s Known on This Subject:

Clinical practice guidelines are used to influence the provider’s care of patients. Implementation of high-quality guidelines can improve care. There have been anecdotal reports of differences between guidelines written on the same condition, but this has never been quantified.

What This Study Adds:

This is the first attempt to quantify the differences between guideline recommendations for the same condition. Overall, there was less agreement between guideline recommendations for asthma than for bronchiolitis.

The creators of guidelines attempt to refine clinical questions and balance the trade-offs of the benefits versus risks of an intervention and its alternatives to influence a clinician’s care of a patient.1 The implementation of clinical practice guidelines can promote high-value care by improving outcomes and reducing costs.2,3 For example, an appropriate decline in the unnecessary use of chest radiographs, steroids, and bronchodilators was observed after the 2006 American Academy of Pediatrics (AAP) bronchiolitis guideline publication.4 However, the authors of a number of studies have demonstrated that differences occur across clinical practice guidelines developed for the same condition.5,8 These result from differences in guideline development, reporting, methodological quality, and content.9,14 These discrepancies can cause confusion about the best treatment for the patient, and naivety about the underlying reason for such differences could lead clinicians to inaccurately apply these recommendations in practice.9 A common means of comparing guidelines is by using quality ratings like the Appraisal of Guidelines Research and Evaluation II (AGREE II).15 However, little is known about potential guideline treatment recommendation agreement among common prevalent pediatric conditions.

Asthma and bronchiolitis are among the most prevalent and costly pediatric medical conditions requiring hospitalization; accordingly, these conditions have been identified as high priorities for research because of their prevalence and cost.16 The objective of this study was to assess the concordance of recommendations for these conditions. Specifically, we aimed to assess the degree of agreement among similar treatment recommendations across different national and international guidelines for asthma and bronchiolitis. We hypothesized that there would be a high level of agreement among similar treatment recommendations across these guidelines.

We performed a literature search to find guidelines for asthma and bronchiolitis by using 4 large guideline databases: the Guidelines International Network, the National Guidelines Clearinghouse, the Canadian Agency for Drugs and Technologies in Health Grey Matters, and the Trip database.17,20 This gray literature search was conducted from December 16 to 17, 2014, (asthma) and on January 9, 2015, (bronchiolitis). Duplicates were removed and the primary author (L.A.B.) screened titles for relevant guidelines.

Guidelines for the treatment of asthma and bronchiolitis published within the last 12 years (January 2003–January 2015) from the 34 countries currently participating in the Organization for Economic Cooperation and Development were included.21 Guideline eligibility criteria are shown in Table 1.

TABLE 1

Guideline Inclusion and Exclusion Criteria

Guideline Inclusion CriteriaGuideline Exclusion Criteria
Asthma guidelines Regional 
Bronchiolitis guidelines Focused on surgical treatment 
Published since 2003 Focused on subspecialist care 
Guideline from an OECD country20  Specific to critical (neonatal or pediatric) 
Key recommendations identified in the guideline Focused on allied health care professionals (nurses, respiratory therapists, etc) 
If more than 1 guideline on the topic was identified by the same guideline group, the most recent version was selected Adult-focused 
 Patient education–focused 
Focused on care not routinely performed by an allopathic physician (ie, acupuncture) 
Symptom- rather than condition-focused in the title (cough versus asthma) 
Written in a language other than English 
Guideline Inclusion CriteriaGuideline Exclusion Criteria
Asthma guidelines Regional 
Bronchiolitis guidelines Focused on surgical treatment 
Published since 2003 Focused on subspecialist care 
Guideline from an OECD country20  Specific to critical (neonatal or pediatric) 
Key recommendations identified in the guideline Focused on allied health care professionals (nurses, respiratory therapists, etc) 
If more than 1 guideline on the topic was identified by the same guideline group, the most recent version was selected Adult-focused 
 Patient education–focused 
Focused on care not routinely performed by an allopathic physician (ie, acupuncture) 
Symptom- rather than condition-focused in the title (cough versus asthma) 
Written in a language other than English 

OECD, Organization for Economic Cooperation and Development.

Extracted data included the recommendation, guideline, disease, the primary outcome of treatment recommendation, and the AGREE II instrument rating to assess guideline quality and reporting, country of origin, and year of guideline publication.

Three authors (L.A.B., J.E., K.L.) independently extracted the data by using structured data collection forms. First, a single guideline was reviewed and scored by all 3 authors, and all discrepancies among the 3 authors were resolved through discussion. Second, all subsequent guidelines were reviewed and data were extracted from them by 2 authors independently. Differences in data extractions were discussed and, if necessary, a third author was used for arbitration. All fields were discussed for unanimous agreement, with the exception of guideline scoring using the AGREE II instrument to assess guideline quality and reporting. This tool has 2 overall guideline assessments and 23 individual questions that fall within 6 different domains: scope and purpose, stakeholder involvement, rigor of development, clarity of presentation, applicability, and editorial independence.15 Discrepancies of >2 points for items on AGREE II were re-reviewed collectively by 2 authors.

In each guideline, recommendations focused on treatment were identified. Excluded recommendations were on assessment, emergency referral criteria, presentation, diagnostic testing, follow-up, prophylaxis, prevention, and education. Only the key recommendations were included, as described by the AGREE II instrument.15 

The primary outcome was treatment recommendation. For each guideline, treatment recommendations were categorized as (1) recommend for: recommendation in favor of an intervention; (2) optional: the intervention was an option; (3) abstain: no recommendation either for or against an intervention; (4) recommend against: a particular treatment was not recommended; or (5) not addressed: the guideline did not specifically address whether to recommend an intervention. The recommendation designation systems that informed this work were the AAP’s policy statement on classifying recommendations and the Grading of Recommendations Assessment, Development and Evaluation system.22,23 Though the instrument was not validated, 2 team members gave the treatment recommendation designation and were checked for consistency.

After collection of the data items, the key recommendations for each guideline were organized by topic to allow comparison of the recommendations among guidelines.

We summarized the overall number of recommendations made for each condition, the frequency of each of the categories of the primary outcome for asthma and bronchiolitis, and the number of recommendations that were not addressed for asthma and bronchiolitis. We compared the guidelines on their quality by using the AGREE II tool. For reference, the evidence strength was reported when available for the key recommendations for the US guidelines for asthma24 and bronchiolitis.23 

We used Cohen’s κ statistic to assess agreement among similar recommendations.25 We used both unweighted and weighted κ in cases in which the primary outcome was treated as categorical and ordinal, respectively. A weighted κ score is different from a standard unweighted κ score in that it allows weighting of differing categories with varying gravity to take into account the magnitude of disagreement present. Analysis was performed by using the R statistical software (www.r-project.org) (R Development Core Team, R Foundation for Statistical Computing, Vienna, Austria).26 We calculated a pairwise κ between guidelines as well as an overall κ score for all recommendations among all the available guidelines. κ scores were categorized as indicating poor agreement (<0), slight agreement (0–0.2), fair agreement (0.21–0.4), moderate agreement (0.41–0.6), substantial agreement (0.61–0.8), or almost perfect agreement (0.81–1.0).27 Confidence intervals (CIs) were determined by bootstrapping (n = 1000).

Sensitivity analyses were conducted with alternate interpretation of the absence of a reported recommendation. First, recommendations originally categorized as not addressed were recoded and analyzed as “missing data.” Second, recommendations originally categorized as not addressed were recategorized as abstain.

This study was considered exempt by the research ethics boards of the Hospital for Sick Children and the University of Toronto, Toronto, Ontario, Canada.

Of 1381 citations, 473 were duplicates. After initial screening of titles and abstracts, 125 documents were identified for full-text review, and 118 were excluded (Fig 1). Seven asthma guidelines were identified.

FIGURE 1

Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram: asthma.

FIGURE 1

Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram: asthma.

There were 166 recommendation topics, with 69 recommendation topics provided in ≥2 guidelines (Table 2). The mean (SD) number of recommendations per guideline was 28 (16.3). The National Heart, Lung, and Blood Institute (NHLBI) asthma guideline contained the most recommendation topics in common with other guidelines, totaling 44 recommendation topics. The American College of Chest Physicians (ACCP) guideline contained the fewest, with only 5 recommendation topics. There was a mean (SD) of 40.6 (16.5) not addressed recommendation topics per guideline.

TABLE 2

Key Recommendations Given by 2 or More Asthma Guidelines and Primary Outcome of Treatment Recommendation

Strength of evidence for NHLBIaNHLBI 2007GINA 2014AAH 2014CTS 2012CPS 2012SIGN/BTS 2014ACCP 2005
AGREE II overall score (out of a possible 7)b  
Recommendation topic         
 Patient health literacy — Did not address Recommend for Recommend for Did not address Did not address Did not address Did not address 
 Control-based management Recommend for Recommend for Did not address Recommend for Did not address Did not address Did not address 
 Treatment decisions based on patient characteristics, patient preferences, and practical issues (inhaler technique, adherence, cost) Recommend for Recommend for Recommend for Recommend for Did not address Recommend for Did not address 
 First-line therapy: SABA Recommend for Recommend for Recommend for Recommend for Did not address Recommend for Did not address 
 Treating with low-dose ICS extremely effective Recommend for Recommend for Recommend for Did not address Did not address Did not address Did not address 
 With persistent symptoms and/or exacerbations: increase treatment if having symptoms even after starting low-dose ICS, first check inhaler technique — Did not address Optional Did not address Recommend for Did not address Did not address Did not address 
 Adults and adolescents: start combo of ICS and LABA if ICS not enough Recommend for Recommend for Did not address Recommend for Did not address Optional Did not address 
 Adults and adolescents with exacerbations despite other therapies: ICS and LABA as maintenance and reliever versus controller and SABA — Did not address Recommend for Did not address Recommend against Did not address Did not address Did not address 
 Patients 6–11 y: increase corticosteroid dose rather than ICS and LABA — Optional Recommend for Did not address Recommend for Did not address Did not address Did not address 
 (Step down) use least amount of medication for optimal control Recommend for Optional Recommend for Recommend for Did not address Recommend for Did not address 
 Patients >6 y: inhaler skills training — Did not address Recommend for Recommend for Did not address Did not address Recommend for Did not address 
 Patients >12 y: inhaler skills training — Did not address Recommend for Recommend for Did not address Did not address Recommend for Did not address 
 Encourage adherence with controller medication (ICS) even when symptoms infrequent — Recommend for Recommend for Did not address Did not address Did not address Did not address Did not address 
 One or more risk factors for exacerbations: prescribe controller therapy (ICS) — Recommend for Recommend for Optional Did not address Did not address Did not address Did not address 
 One or more risk factor for exacerbation: identify and address modifiable risk factors — Recommend for Recommend for Did not address Recommend for Did not address Did not address Did not address 
 One or more risk factors for exacerbation: consider nonpharmacologic strategies to reduce symptoms — Did not address Optional Did not address Did not address Did not address Optional Did not address 
 Give advice on EIB prevention to those symptomatic — Recommend for Recommend for Did not address Did not address Did not address Did not address Did not address 
 EIB: prescribe controllers if risk factors, symptoms outside exercise — Recommend for Recommend for Did not address Did not address Did not address Did not address Did not address 
 Patients >6 y: refer difficult-to-manage patients to specialists after addressing common treatment problems Recommend for Recommend for Optional Recommend for Recommend for Did not address Did not address 
 Patients 0–5 y: refer difficult-to-manage patients to specialists after addressing common treatment problems Recommend for Did not address Optional Recommend for Recommend for Did not address Did not address 
 With exacerbation, start with repeated doses of SABA (most patients: MDI and spacer) Recommend for Recommend for Recommend for Did not address Recommend for Did not address Recommend for 
 With exacerbation, give oral steroids early — Recommend for Recommend for Recommend for Did not address Did not address Recommend for Recommend for 
 Patients >6 y: with exacerbation, give oral corticosteroids — Recommend for Did not address Optional Recommend for Recommend for Did not address Did not address 
 Patients <6 y: oral corticosteroids — Did not address Recommend for Recommend against Did not address Recommend for Did not address Did not address 
 Patients 0–5 y: with exacerbation, give oxygen as needed — Did not address Recommend for Recommend for Did not address Recommend for Did not address Did not address 
 Patients >6 y: with exacerbation, give oxygen as needed — Recommend for Recommend for Recommend for Did not address Recommend for Recommend for Did not address 
 With severe exacerbation, ipratropium bromide Recommend for Recommend for Recommend for Did not address Did not address Did not address Did not address 
 With severe exacerbation, IV magnesium sulfate Optional Optional Optional Did not address Did not address Recommend for Did not address 
 Patients 6–11 y: after exacerbation, start controller (ICS) or step up dose for 2–4 wk — Did not address Recommend for Optional Recommend against Did not address Did not address Did not address 
 Patients >12 y: after exacerbation, start controller (ICS) or step up dose for 2–4 wk — Did not address Recommend for Optional Recommend for Did not address Did not address Did not address 
 Patients >6 y: antibiotics — Recommend against Recommend against Recommend against Did not address Did not address Did not address Did not address 
 All ages: antibiotics — Recommend against Did not address Recommend against Did not address Did not address Did not address Did not address 
 Patients 0–5 y: treat wheezing in children with SABA — Did not address Recommend for Recommend for Did not address Did not address Did not address Did not address 
 Trial controller therapy (ICS) for children with frequent or severe wheezing Recommend for Recommend for Optional Did not address Did not address Did not address Did not address 
 Patients 0–5 y: choice of inhaler device in kids by age and capability; preferred device: MDI + spacer with mask or mouthpiece Recommend for Recommend for Recommend for Did not address Did not address Recommend for Did not address 
 Patients >5 y: choice of inhaler device in kids by age and capability; preferred device MDI + spacer with mask or mouthpiece Recommend for Did not address Recommend for Recommend for Recommend for Recommend for Optional 
 Exacerbation: home use of SABA — Did not address Recommend for Recommend for Did not address Did not address Did not address Did not address 
 Exacerbation: parent-initiated oral corticosteroids — Did not address Optional Recommend against Recommend against Did not address Did not address Did not address 
 With exacerbation, dosing of SABA (2–6 puffs every 20 min for first h) — Did not address Recommend for Recommend for Did not address Did not address Did not address Did not address 
 Patients 0–5 y: oral prednisone — Did not address Recommend for Recommend for Did not address Recommend for Did not address Did not address 
 Patients >6 y: oral prednisone — Did not address Did not address Recommend for Recommend for Recommend for Did not address Did not address 
 Patients >12 y: oral prednisone — Did not address Did not address Recommend for Recommend for Recommend for Did not address Did not address 
 Patients >2 y: intermittent or mild persistent: LTRA (montelukast) as first choice controller medicine Optional Recommend against Optional Did not address Did not address Recommend for Did not address 
 Patients >2 y :moderate to severe persistent symptoms, ICS as first-choice controller medicine Recommend for Recommend for Optional Did not address Did not address Did not address Did not address 
 Patients <2 y: sodium cromoglycate Optional Did not address Optional Did not address Did not address Did not address Did not address 
 Patients >2 y: sodium cromoglycate Optional Did not address Optional Did not address Did not address Did not address Did not address 
 Patients 5–11 y: regular use of theophylline Optional Did not address Recommend against Did not address Did not address Did not address Did not address 
 Ipratropium for regular use Optional Did not address Recommend against Did not address Did not address Did not address Did not address 
 Patients <6 y: oral corticosteroids with severe exacerbation — Did not address Did not address Recommend for Did not address Did not address Optional Did not address 
 Needing β2 agonist >2 × per wk: prescribe controller therapy (ICS) Recommend for Did not address Optional Did not address Did not address Did not address Did not address 
 Omalizumab Recommend for Did not address Optional Did not address Did not address Did not address Did not address 
 Cleaning spacer — Did not address Did not address Recommend for Did not address Recommend for Did not address Did not address 
 All children: precautions with inhaled corticosteroids (ICS) Recommend for Did not address Recommend for Did not address Did not address Did not address Did not address 
 All ages: precautions with inhaled corticosteroids (ICS) Recommend for Did not address Recommend for Did not address Did not address Did not address Did not address 
 Nebulizer — Did not address Did not address Optional Did not address Did not address Optional Recommend for 
 Patients: demonstrate technique Recommend for Did not address Recommend for Did not address Recommend for Did not address Did not address 
 With severe exacerbation, intermittent nebulizer SABA — Did not address Did not address Recommend for Did not address Did not address Did not address Recommend for 
 With life-threatening asthma, continuous nebulizer of SABA — Did not address Did not address Recommend for Did not address Did not address Did not address Recommend for 
 Add-on therapy to salbutamol (SABA) — Did not address Did not address Optional Did not address Optional Did not address Did not address 
 IV salbutamol dosing guide — Did not address Did not address Recommend for Did not address Did not address Did not address Did not address 
 Uncontrolled with medium dose ICS: add adjunctive therapy (usually LABA) Recommend for Did not address Did not address Recommend for Did not address Optional Did not address 
 Patients >12 y: third-line option: LTRA or increase ICS Optional Did not address Did not address Optional Did not address Did not address Did not address 
 Patients >16 y: ICS and/or LABA as reliever for mild intermittent asthma not on controller Recommend against Did not address Did not address Recommend against Did not address Did not address Did not address 
 Patients <16 y: ICS and/or LABA as reliever when not on controller therapy Recommend against Did not address Did not address Optional Did not address Did not address Did not address 
 Children and adults: ICS and/or LABA as reliever when on ICS monotherapy Recommend against Did not address Did not address Abstain Did not address Did not address Did not address 
 Mild persistent asthma: daily ICS versus intermittent ICS Recommend for Did not address Did not address Recommend for Did not address Did not address Did not address 
 Patients <12 y: ICS and/or LABA adjustable maintenance dosing versus increasing ICS adjustable maintenance dosing — Did not address Did not address Did not address Abstain Did not address Recommend against Did not address 
 ICS and/or LABA AMD versus increased ICS dose — Did not address Did not address Did not address Abstain Recommend for Recommend against Did not address 
 LABA as monotherapy — Recommend against Did not address Did not address Recommend against Did not address Did not address Did not address 
 ICS effective Recommend for Did not address Did not address Did not address Did not address Recommend for Did not address 
Strength of evidence for NHLBIaNHLBI 2007GINA 2014AAH 2014CTS 2012CPS 2012SIGN/BTS 2014ACCP 2005
AGREE II overall score (out of a possible 7)b  
Recommendation topic         
 Patient health literacy — Did not address Recommend for Recommend for Did not address Did not address Did not address Did not address 
 Control-based management Recommend for Recommend for Did not address Recommend for Did not address Did not address Did not address 
 Treatment decisions based on patient characteristics, patient preferences, and practical issues (inhaler technique, adherence, cost) Recommend for Recommend for Recommend for Recommend for Did not address Recommend for Did not address 
 First-line therapy: SABA Recommend for Recommend for Recommend for Recommend for Did not address Recommend for Did not address 
 Treating with low-dose ICS extremely effective Recommend for Recommend for Recommend for Did not address Did not address Did not address Did not address 
 With persistent symptoms and/or exacerbations: increase treatment if having symptoms even after starting low-dose ICS, first check inhaler technique — Did not address Optional Did not address Recommend for Did not address Did not address Did not address 
 Adults and adolescents: start combo of ICS and LABA if ICS not enough Recommend for Recommend for Did not address Recommend for Did not address Optional Did not address 
 Adults and adolescents with exacerbations despite other therapies: ICS and LABA as maintenance and reliever versus controller and SABA — Did not address Recommend for Did not address Recommend against Did not address Did not address Did not address 
 Patients 6–11 y: increase corticosteroid dose rather than ICS and LABA — Optional Recommend for Did not address Recommend for Did not address Did not address Did not address 
 (Step down) use least amount of medication for optimal control Recommend for Optional Recommend for Recommend for Did not address Recommend for Did not address 
 Patients >6 y: inhaler skills training — Did not address Recommend for Recommend for Did not address Did not address Recommend for Did not address 
 Patients >12 y: inhaler skills training — Did not address Recommend for Recommend for Did not address Did not address Recommend for Did not address 
 Encourage adherence with controller medication (ICS) even when symptoms infrequent — Recommend for Recommend for Did not address Did not address Did not address Did not address Did not address 
 One or more risk factors for exacerbations: prescribe controller therapy (ICS) — Recommend for Recommend for Optional Did not address Did not address Did not address Did not address 
 One or more risk factor for exacerbation: identify and address modifiable risk factors — Recommend for Recommend for Did not address Recommend for Did not address Did not address Did not address 
 One or more risk factors for exacerbation: consider nonpharmacologic strategies to reduce symptoms — Did not address Optional Did not address Did not address Did not address Optional Did not address 
 Give advice on EIB prevention to those symptomatic — Recommend for Recommend for Did not address Did not address Did not address Did not address Did not address 
 EIB: prescribe controllers if risk factors, symptoms outside exercise — Recommend for Recommend for Did not address Did not address Did not address Did not address Did not address 
 Patients >6 y: refer difficult-to-manage patients to specialists after addressing common treatment problems Recommend for Recommend for Optional Recommend for Recommend for Did not address Did not address 
 Patients 0–5 y: refer difficult-to-manage patients to specialists after addressing common treatment problems Recommend for Did not address Optional Recommend for Recommend for Did not address Did not address 
 With exacerbation, start with repeated doses of SABA (most patients: MDI and spacer) Recommend for Recommend for Recommend for Did not address Recommend for Did not address Recommend for 
 With exacerbation, give oral steroids early — Recommend for Recommend for Recommend for Did not address Did not address Recommend for Recommend for 
 Patients >6 y: with exacerbation, give oral corticosteroids — Recommend for Did not address Optional Recommend for Recommend for Did not address Did not address 
 Patients <6 y: oral corticosteroids — Did not address Recommend for Recommend against Did not address Recommend for Did not address Did not address 
 Patients 0–5 y: with exacerbation, give oxygen as needed — Did not address Recommend for Recommend for Did not address Recommend for Did not address Did not address 
 Patients >6 y: with exacerbation, give oxygen as needed — Recommend for Recommend for Recommend for Did not address Recommend for Recommend for Did not address 
 With severe exacerbation, ipratropium bromide Recommend for Recommend for Recommend for Did not address Did not address Did not address Did not address 
 With severe exacerbation, IV magnesium sulfate Optional Optional Optional Did not address Did not address Recommend for Did not address 
 Patients 6–11 y: after exacerbation, start controller (ICS) or step up dose for 2–4 wk — Did not address Recommend for Optional Recommend against Did not address Did not address Did not address 
 Patients >12 y: after exacerbation, start controller (ICS) or step up dose for 2–4 wk — Did not address Recommend for Optional Recommend for Did not address Did not address Did not address 
 Patients >6 y: antibiotics — Recommend against Recommend against Recommend against Did not address Did not address Did not address Did not address 
 All ages: antibiotics — Recommend against Did not address Recommend against Did not address Did not address Did not address Did not address 
 Patients 0–5 y: treat wheezing in children with SABA — Did not address Recommend for Recommend for Did not address Did not address Did not address Did not address 
 Trial controller therapy (ICS) for children with frequent or severe wheezing Recommend for Recommend for Optional Did not address Did not address Did not address Did not address 
 Patients 0–5 y: choice of inhaler device in kids by age and capability; preferred device: MDI + spacer with mask or mouthpiece Recommend for Recommend for Recommend for Did not address Did not address Recommend for Did not address 
 Patients >5 y: choice of inhaler device in kids by age and capability; preferred device MDI + spacer with mask or mouthpiece Recommend for Did not address Recommend for Recommend for Recommend for Recommend for Optional 
 Exacerbation: home use of SABA — Did not address Recommend for Recommend for Did not address Did not address Did not address Did not address 
 Exacerbation: parent-initiated oral corticosteroids — Did not address Optional Recommend against Recommend against Did not address Did not address Did not address 
 With exacerbation, dosing of SABA (2–6 puffs every 20 min for first h) — Did not address Recommend for Recommend for Did not address Did not address Did not address Did not address 
 Patients 0–5 y: oral prednisone — Did not address Recommend for Recommend for Did not address Recommend for Did not address Did not address 
 Patients >6 y: oral prednisone — Did not address Did not address Recommend for Recommend for Recommend for Did not address Did not address 
 Patients >12 y: oral prednisone — Did not address Did not address Recommend for Recommend for Recommend for Did not address Did not address 
 Patients >2 y: intermittent or mild persistent: LTRA (montelukast) as first choice controller medicine Optional Recommend against Optional Did not address Did not address Recommend for Did not address 
 Patients >2 y :moderate to severe persistent symptoms, ICS as first-choice controller medicine Recommend for Recommend for Optional Did not address Did not address Did not address Did not address 
 Patients <2 y: sodium cromoglycate Optional Did not address Optional Did not address Did not address Did not address Did not address 
 Patients >2 y: sodium cromoglycate Optional Did not address Optional Did not address Did not address Did not address Did not address 
 Patients 5–11 y: regular use of theophylline Optional Did not address Recommend against Did not address Did not address Did not address Did not address 
 Ipratropium for regular use Optional Did not address Recommend against Did not address Did not address Did not address Did not address 
 Patients <6 y: oral corticosteroids with severe exacerbation — Did not address Did not address Recommend for Did not address Did not address Optional Did not address 
 Needing β2 agonist >2 × per wk: prescribe controller therapy (ICS) Recommend for Did not address Optional Did not address Did not address Did not address Did not address 
 Omalizumab Recommend for Did not address Optional Did not address Did not address Did not address Did not address 
 Cleaning spacer — Did not address Did not address Recommend for Did not address Recommend for Did not address Did not address 
 All children: precautions with inhaled corticosteroids (ICS) Recommend for Did not address Recommend for Did not address Did not address Did not address Did not address 
 All ages: precautions with inhaled corticosteroids (ICS) Recommend for Did not address Recommend for Did not address Did not address Did not address Did not address 
 Nebulizer — Did not address Did not address Optional Did not address Did not address Optional Recommend for 
 Patients: demonstrate technique Recommend for Did not address Recommend for Did not address Recommend for Did not address Did not address 
 With severe exacerbation, intermittent nebulizer SABA — Did not address Did not address Recommend for Did not address Did not address Did not address Recommend for 
 With life-threatening asthma, continuous nebulizer of SABA — Did not address Did not address Recommend for Did not address Did not address Did not address Recommend for 
 Add-on therapy to salbutamol (SABA) — Did not address Did not address Optional Did not address Optional Did not address Did not address 
 IV salbutamol dosing guide — Did not address Did not address Recommend for Did not address Did not address Did not address Did not address 
 Uncontrolled with medium dose ICS: add adjunctive therapy (usually LABA) Recommend for Did not address Did not address Recommend for Did not address Optional Did not address 
 Patients >12 y: third-line option: LTRA or increase ICS Optional Did not address Did not address Optional Did not address Did not address Did not address 
 Patients >16 y: ICS and/or LABA as reliever for mild intermittent asthma not on controller Recommend against Did not address Did not address Recommend against Did not address Did not address Did not address 
 Patients <16 y: ICS and/or LABA as reliever when not on controller therapy Recommend against Did not address Did not address Optional Did not address Did not address Did not address 
 Children and adults: ICS and/or LABA as reliever when on ICS monotherapy Recommend against Did not address Did not address Abstain Did not address Did not address Did not address 
 Mild persistent asthma: daily ICS versus intermittent ICS Recommend for Did not address Did not address Recommend for Did not address Did not address Did not address 
 Patients <12 y: ICS and/or LABA adjustable maintenance dosing versus increasing ICS adjustable maintenance dosing — Did not address Did not address Did not address Abstain Did not address Recommend against Did not address 
 ICS and/or LABA AMD versus increased ICS dose — Did not address Did not address Did not address Abstain Recommend for Recommend against Did not address 
 LABA as monotherapy — Recommend against Did not address Did not address Recommend against Did not address Did not address Did not address 
 ICS effective Recommend for Did not address Did not address Did not address Did not address Recommend for Did not address 

AMD, adjustable maintenance dosing; MDI, metered dose inhaler;—, not reported.

a

The level of evidence is given for the NHLBI guideline in letters per the system of evidence reporting in the NHLBI guideline when it was reported.24 Evidence Category A: RCTs, rich body of data; Evidence Category B: RCTs, limited body of data; Evidence Category C: nonrandomized trials and observational studies; Evidence Category D: panel consensus judgment.

b

AGREE II scoring instrument to assess guideline quality and reporting: this tool has overall guideline assessments (overall score above), as well as 23 individual questions.

The AGREE II overall quality score (total score of 7) ranged from 3 to 6 points. The Scottish Intercollegiate Guidelines Network (SIGN) and Canadian Thoracic Society (CTS) guidelines had the best overall AGREE II score of 6, and the Canadian Paediatric Society (CPS) guideline had the lowest score of 3 (Table 2).

The overall unweighted and weighted κ scores were both 0.03 (95% CIs: −0.01 to 0.07 and −0.01 to 0.10, respectively). Both scores signify only slight agreement (Table 3). The agreement between guideline pairs was poor (Australian Asthma Handbook (AAH) and CTS, unweighted κ score: −0.15 [95% CI: −0.28 to −0.02]; weighted κ score: −0.2 [95% CI: −0.04 to −0.01]) to fair (AAH and CPS, unweighted κ score: 0.18 [95% CI: 0.07 to 0.29]; weighted κ score: 0.24 [95% CI: 0.1 to 0.39]).

TABLE 3

Pairwise and Overall κ Estimates for Asthma Guidelines

Weighted
 GINA 0.13 (−0.1 to 0.36) −0.03 (−0.25 to 0.18) −0.06 (−0.25 to 0.13) 0.11 (−0.07 to 0.3) 0.09 (−0.15 to 0.32) −0.08 (−0.21 to 0.04) 
Unweighted 0.12 (−0.03 to 0.28) AAH −0.2 (−0.4 to −0.01) 0.24 (0.1 to 0.39) 0.15 (−0.02 to 0.31) −0.05 (−0.29 to 0.18) 0.10 (0.01 to 0.19) 
−0.04 (−0.2 to 0.11) −0.15 (−0.28 to −0.02) CTS 0.15 (−0.11 to 0.41) 0.07 (−0.18 to 0.32) 0.15 (−0.06 to 0.35) −0.07 (−0.21 to 0.08) 
−0.04 (−0.21 to 0.12) 0.18 (0.07 to 0.29) 0.10 (−0.9 to 0.30) CPS −0.11 (−0.33 to 0.11) −0.02 (−0.21 to 0.17) 0.08 (−0.14 to 0.3) 
0.09 (−0.06 to 0.25) 0.07 (−0.06 to 0.2) 0.04 (−0.13 to 0.2) −0.08 (−0.26 to 0.09) SIGN/BTS 0.13 (0.05 to 0.31) 0.03 (−0.15 to 0.21) 
0.11 (−0.07 to 0.28) 0.06 (−0.11 to 0.22) 0.17 (0.01 to 0.33) −0.02 (−0.16 to 0.13) 0.04 (−0.09 to 0.17) NHLBI/NAEPP −0.05 (−0.17 to 0.07) 
−0.07 (−0.17 to 0.04) 0.06 (−0.01 to 0.12) −0.07 (−0.16 to 0.02) 0.07 (−0.13 to 0.27) −0.01 (−0.11 to 0.09) −0.05 (−0.14 to 0.04) ACCP/ACAAI 
 Overall κ unweighted 0.03 (−0.01 to 0.07)  
Overall κ weighted 0.03 (−0.01 to 0.10)  
Weighted
 GINA 0.13 (−0.1 to 0.36) −0.03 (−0.25 to 0.18) −0.06 (−0.25 to 0.13) 0.11 (−0.07 to 0.3) 0.09 (−0.15 to 0.32) −0.08 (−0.21 to 0.04) 
Unweighted 0.12 (−0.03 to 0.28) AAH −0.2 (−0.4 to −0.01) 0.24 (0.1 to 0.39) 0.15 (−0.02 to 0.31) −0.05 (−0.29 to 0.18) 0.10 (0.01 to 0.19) 
−0.04 (−0.2 to 0.11) −0.15 (−0.28 to −0.02) CTS 0.15 (−0.11 to 0.41) 0.07 (−0.18 to 0.32) 0.15 (−0.06 to 0.35) −0.07 (−0.21 to 0.08) 
−0.04 (−0.21 to 0.12) 0.18 (0.07 to 0.29) 0.10 (−0.9 to 0.30) CPS −0.11 (−0.33 to 0.11) −0.02 (−0.21 to 0.17) 0.08 (−0.14 to 0.3) 
0.09 (−0.06 to 0.25) 0.07 (−0.06 to 0.2) 0.04 (−0.13 to 0.2) −0.08 (−0.26 to 0.09) SIGN/BTS 0.13 (0.05 to 0.31) 0.03 (−0.15 to 0.21) 
0.11 (−0.07 to 0.28) 0.06 (−0.11 to 0.22) 0.17 (0.01 to 0.33) −0.02 (−0.16 to 0.13) 0.04 (−0.09 to 0.17) NHLBI/NAEPP −0.05 (−0.17 to 0.07) 
−0.07 (−0.17 to 0.04) 0.06 (−0.01 to 0.12) −0.07 (−0.16 to 0.02) 0.07 (−0.13 to 0.27) −0.01 (−0.11 to 0.09) −0.05 (−0.14 to 0.04) ACCP/ACAAI 
 Overall κ unweighted 0.03 (−0.01 to 0.07)  
Overall κ weighted 0.03 (−0.01 to 0.10)  

Pairwise κ estimates between 2 guidelines and overall κ estimate with the corresponding 95% CIs. The values below the diagonal are unweighted κ scores, and the values above the diagonal are weighted κ scores. N = 69 recommendations used to calculate the estimates; CIs were calculated by using bootstrapping, n = 1000. The GINA guidelines were published 2014; the AAH guidelines were published in 2014; The CTS guidelines were published in 2012; the CPS guidelines were published in 2012; the SIGN and BTS guidelines were published in 2014; the NHLBI and NAEPP guidelines were published in 2007, and the ACCP and American College of Asthma, Allergy, and Immunology guidelines were published in 2005. ACAAI, American College of Asthma, Allergy, and Immunology; BTS, British Thoracic Society. NAEPP, National Asthma Education and Prevention Program.

Sensitivity Analysis 1

When recommendations originally categorized as not addressed were recoded and analyzed as missing data, the κ analysis (both overall and paired) could not be completed because of the large number of missing values. The key recommendation topics from the 7 guidelines on asthma were discrepant. In 31 instances, only 2 guidelines contained recommendations that could be compared.

Sensitivity Analysis 2

When recommendations originally categorized as not addressed were recategorized as abstain, overall, the weighted and unweighted κ scores showed slight agreement (overall unweighted κ: 0.04 [95% CI: 0 to 0.08]; overall weighted κ: 0.12 (95% CI: 0.06 to 0.19]) (Supplemental Table 6). The pairwise agreement ranged from poor to fair, similarly to the primary analysis, although some differences in guideline pairs were noted.

Of 322 citations, 65 were duplicates. After initial screening of titles and abstracts, 13 documents were identified and the full texts were obtained. Nine guidelines were excluded after obtaining the full texts (Fig 2). Four bronchiolitis guidelines were identified.

FIGURE 2

Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram: bronchiolitis.

FIGURE 2

Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram: bronchiolitis.

There were 46 recommendation topics, with 21 recommendation topics provided in ≥2 guidelines (Table 4). The mean (SD) number of recommendations per guideline was 15 (2.7). The SIGN bronchiolitis guideline contained the fewest recommendation topics in common with other guidelines, totaling 13 recommendation topics. The recommendations included in the Spanish National Health System (SNHS) guideline were all addressed in other guidelines as well. There was a mean (SD) of 6 (2.7) not addressed recommendation topics per guideline.

TABLE 4

Key Recommendations Given by 2 or More Bronchiolitis Guidelines and Primary Outcome of Treatment Recommendation

Strength of evidence for the AAPaAAP 2014SIGN 2006SNHS 2010CPS 2014
AGREE II overall score (out of a possible 7)b  
Recommendation topics      
 Routine use of β agonists Recommend against Recommend against Recommend against Recommend against 
 Routine use of α agonists Recommend against Recommend against Recommend against Recommend against 
 Trial of α or β adrenergic medication — Did not address Did not address Optional Optional 
 Corticosteroid medications (oral or inhaled) Recommend against Recommend against Recommend against Recommend against 
 Corticosteroids: ventilated patients Recommend against Did not address Did not address Did not address 
 Ribavirin — Did not address Recommend against Recommend against Did not address 
 Antibiotics for bacterial coinfection Recommend for Did not address Recommend for Recommend for 
 Antibiotics for bronchiolitis Recommend against Recommend against Recommend against Recommend against 
 Hydration assessment and ability to take oral fluids — Did not address Did not address Recommend for Recommend for 
 Chest physiotherapy Recommend against Recommend against Recommend against Recommend against 
 Supplemental O2 use <90% Recommend for Recommend for Recommend for Recommend for 
 Supplemental O2 use <92% Optional Recommend for Recommend for Did not address 
 O2-using continuous pulse-oximetry Optional Did not address Did not address Optional 
 3% hypertonic saline trial Optional Did not address Recommend for Optional 
 Hypertonic saline in emergency department Recommend against Did not address Did not address Recommend against 
 Montelukast — Did not address Abstain Recommend against Did not address 
 Respiratory secretion aspiration — Did not address Recommend for Recommend for Optional 
 Feeds, nasogastric Recommend for Recommend for Optional Did not address 
 Feeds, intravenous hydration Recommend for Did not address Recommend for Recommend for 
 Noninvasive ventilation — Did not address Recommend for Recommend for Did not address 
 Invasive ventilation — Did not address Recommend for Recommend for Did not address 
Strength of evidence for the AAPaAAP 2014SIGN 2006SNHS 2010CPS 2014
AGREE II overall score (out of a possible 7)b  
Recommendation topics      
 Routine use of β agonists Recommend against Recommend against Recommend against Recommend against 
 Routine use of α agonists Recommend against Recommend against Recommend against Recommend against 
 Trial of α or β adrenergic medication — Did not address Did not address Optional Optional 
 Corticosteroid medications (oral or inhaled) Recommend against Recommend against Recommend against Recommend against 
 Corticosteroids: ventilated patients Recommend against Did not address Did not address Did not address 
 Ribavirin — Did not address Recommend against Recommend against Did not address 
 Antibiotics for bacterial coinfection Recommend for Did not address Recommend for Recommend for 
 Antibiotics for bronchiolitis Recommend against Recommend against Recommend against Recommend against 
 Hydration assessment and ability to take oral fluids — Did not address Did not address Recommend for Recommend for 
 Chest physiotherapy Recommend against Recommend against Recommend against Recommend against 
 Supplemental O2 use <90% Recommend for Recommend for Recommend for Recommend for 
 Supplemental O2 use <92% Optional Recommend for Recommend for Did not address 
 O2-using continuous pulse-oximetry Optional Did not address Did not address Optional 
 3% hypertonic saline trial Optional Did not address Recommend for Optional 
 Hypertonic saline in emergency department Recommend against Did not address Did not address Recommend against 
 Montelukast — Did not address Abstain Recommend against Did not address 
 Respiratory secretion aspiration — Did not address Recommend for Recommend for Optional 
 Feeds, nasogastric Recommend for Recommend for Optional Did not address 
 Feeds, intravenous hydration Recommend for Did not address Recommend for Recommend for 
 Noninvasive ventilation — Did not address Recommend for Recommend for Did not address 
 Invasive ventilation — Did not address Recommend for Recommend for Did not address 

Key recommendations given by each guideline for comparison on whether a treatment was recommended. AGREE II scoring instrument to assess guideline quality and reporting: overall score reported. The level of evidence is given in letters per the system of evidence reporting in the AAP guideline. —, not available.

a

The level of evidence is given in letters per the system of evidence reporting in the AAP guideline.22 Evidence Level A: Intervention: well-designed and conducted trials, meta-analyses on applicable populations; Diagnosis: independent gold standard studies of applicable populations. Evidence Level B: trials or diagnostic studies with minor limitations; consistent findings from multiple observational studies. Evidence Level C: single or few observational studies or multiple studies with inconsistent findings or major limitations. Evidence Level D: expert opinion, case reports, reasoning from first principles. Evidence Level X: exceptional situations in which validating studies cannot be performed and there is a clear preponderance of benefit or harm.

b

AGREE II scoring instrument to assess guideline quality and reporting: this tool has overall guideline assessments (overall score above), as well as 23 individual questions.

The overall AGREE II quality score (total score of 7) ranged from 2 to 6. The best AGREE II score was the SNHS guideline score of 6, and the CPS guideline received the lowest score of 2 (Table 4).

The overall unweighted κ score for the bronchiolitis treatment recommendations demonstrated fair agreement (0.32 [95% CI: 0.16 to 0.52]), and the overall weighted κ score signified slight agreement (0.15 [95% CI: −0.01 to 0.5]) (Table 5). There was slight agreement (SIGN and the CPS, unweighted κ score: 0.1 [95% CI: −0.17 to 0.36]; weighted κ score: −0.35 [95% CI: −0.79 to 0.09]) to moderate agreement (AAP and CPS, unweighted κ score: 0.61 [95% CI: 0.35 to 0.87]; weighted κ score: 0.39 [95% CI: −0.09 to 0.87]; SNHS and SIGN, weighted κ score: 0.39 [95% CI: 0.02 to 0.75]) between guideline pairs.

TABLE 5

Pairwise and Overall κ Estimates for Bronchiolitis Guidelines

Weighted
 AAP 0 (−0.49 to 0.5) 0.15 (−0.2 to 0.5) 0.39 (−0.09 to 0.87) 
Unweighted 0.21 (−0.07 to 0.49) SIGN 0.39 (0.02 to 0.75) −0.35 (−0.79 to 0.09) 
0.23 (0.01 to 0.46) 0.48 (0.24 to 0.72) SNHS 0.3 (−0.06 to 0.66) 
0.61 (0.35 to 0.87) 0.1 (−0.17 to 0.36) 0.31 (0.07 to 0.55) CPS 
 Overall κ unweighted 0.32 (0.16 to 0.52)  
 Overall κ weighted 0.15 (−0.01 to 0.5)  
Weighted
 AAP 0 (−0.49 to 0.5) 0.15 (−0.2 to 0.5) 0.39 (−0.09 to 0.87) 
Unweighted 0.21 (−0.07 to 0.49) SIGN 0.39 (0.02 to 0.75) −0.35 (−0.79 to 0.09) 
0.23 (0.01 to 0.46) 0.48 (0.24 to 0.72) SNHS 0.3 (−0.06 to 0.66) 
0.61 (0.35 to 0.87) 0.1 (−0.17 to 0.36) 0.31 (0.07 to 0.55) CPS 
 Overall κ unweighted 0.32 (0.16 to 0.52)  
 Overall κ weighted 0.15 (−0.01 to 0.5)  

Pairwise κ estimates between 2 guidelines and overall κ estimate with the corresponding 95% CIs. The values below the diagonal are unweighted κ scores, and the values above the diagonal are weighted κ scores. N = 21 recommendations used to calculate the estimates; CIs were calculated by using bootstrapping, n = 1000. The AAP guidelines were published in 2014; the CPS guidelines were published in 2014, the SIGN guidelines were published in 2006; the SNHS guidelines were published in 2010.

Sensitivity Analysis 1

When recommendations originally categorized as not addressed were recoded and analyzed as missing data, this substantially changed the κ scores (Supplemental Table 7). The overall unweighted κ score indicated substantial agreement (0.75 [95% CI: 0.53 to 0.94]) and the weighted κ score was almost perfect (0.92 [95% CI: 0.82 to 0.99]). For the pairwise comparison, both unweighted and weighted κ scores were between moderate and almost perfect.

Sensitivity Analysis 2

When recommendations originally categorized as not addressed were recategorized as abstain, overall unweighted and weighted κ scores indicated fair (0.34 [95% CI: 0.17 to 0.51]) to substantial (0.61 [95% CI: 0.44 to 0.78]) agreement and the pairs’ agreement demonstrated slight to substantial agreement for unweighted and weighted κ scores (Supplemental Table 8). These results mirrored the results obtained from the primary analysis and may better estimate the true overall κ scores and pairwise agreement, specifically for the weighted scores that are dependent on the ordinal nature of the scale.

This is the first report in the literature in which quantitative methods are used to compare clinical practice guideline treatment recommendations among different national and international guidelines. Focusing on highly prevalent pediatric conditions cared for by pediatricians, we found less agreement than anticipated among national and international guidelines for asthma (Table 2) than for bronchiolitis (Table 4). This is likely because of the large number of not addressed recommendations among the asthma guidelines. In addition, there was a substantial difference in the κ scores when recommendations were categorized as not addressed and when they were considered missing data for both asthma and bronchiolitis. When analyzed in this manner, those recommendations that were not addressed and then recoded as missing data were not accounted for in this analysis, leading to fewer overall comparisons. When the comparisons with the not addressed category were removed from the analysis for bronchiolitis, agreement becomes nearly perfect; however, this may falsely overstate the agreement between guidelines.

Additionally, the difference could also be attributed to the type of treatment recommendations that are being put forth. It may be easier to agree on nonintervention recommendations that are common for bronchiolitis than on recommendations for an appropriate intervention, as often occurs for asthma.

The authors of numerous studies have compared guidelines for the same condition by using qualitative and descriptive analyses.7,10,11,14,28,31 When guidelines have been quantitatively compared in the literature, the main comparison is focused on guideline quality as assessed by using the AGREE II instrument.6,8,32,33 We too found variation in guideline quality by using the AGREE II instrument. We have not found any previous reports in which the differences in agreement across guidelines for the same condition are quantified.

The differences reported in this study are clinically important. In Table 2, it is apparent that there are occasions when the asthma guidelines do agree. However, there are also many instances in which they do not. There are 4 examples in which 1 guideline recommended a treatment and another recommended against the same treatment of asthma. One such example is the use of a leukotriene receptor antagonist (montelukast) for children over 2 years as the first choice of controller therapy for mild persistent asthma. The Global Initiative for Asthma (GINA) recommends against the practice, whereas it was optional in the AAH and NHLBI guidelines and recommended for use in the SIGN guideline. Though there is less variation in the bronchiolitis guidelines (Table 4), there are still discrepancies, although none were as stark as 1 guideline recommending a treatment and another recommending against it. For example, a montelukast use recommendation was abstained from in the SIGN guideline, but montelukast use was not recommended in the SNHS guideline. Overall, these differences can make the treatment of patients confusing when a provider is trying to follow evidence-based clinical practices guidelines.

The AAP develops clinical practice guidelines independently as well as through collaborations with other societies. It also endorses guidelines from other organizations.22 Although the AAP developed its own system for evaluating the evidence and providing recommendations,22 evidence is often insufficient, leaving the AAP guideline panel to make recommendations on the basis of little evidence.34 It will be important for the AAP to be cognizant of the differences between their endorsed guidelines and those of other national and international pediatric societies. This also reveals the continued variability in quality of guidelines in the United States and in other countries that aim to bring the best clinical care through clinical practice guidelines to pediatric patients.34 

There are limitations to this study. First, there was a lack of pairwise comparisons, particularly for asthma. This issue has been termed the “Kappa Paradox” in the statistical literature.35 There were a large number of recommendations that were categorized as not addressed. For example, the CPS guideline had few key recommendations outlined, leading to many recommendation topics being categorized as not addressed. The ACCP guideline had few recommendation topics in common with the other guidelines; this guideline was focused largely on inhalation devices and delivery, leading to many recommendation topics being categorized as not addressed. This lack of data is the most plausible explanation for the differences that we identified in the analysis when the category not addressed was recoded to missing data, leading to fewer available comparisons. Second, although this study was an attempt to quantify the differences between guideline treatment recommendations that previously have been compared qualitatively, this method may not be sensitive enough to the subtle semantics of recommendations as they are written. As a team, we had many discussions about the differences in language for recommend for, optional, abstain, and recommend against. This categorization may not address the subtleties of language that explain in explicit detail the differences between guidelines. For instance, there is a 22-page chapter dedicated to explaining the differences in the current US and European asthma guidelines.30 Third, there were several limitations to the search and retrieval of guidelines for this study. First, we limited our search to a structured gray literature search that was not peer reviewed by a librarian. However, in comparison with a Medline, Embase, and similar but more limited gray literature search peer reviewed by a librarian for another unpublished study, there were few differences in the number of guidelines retrieved for bronchiolitis and asthma. For bronchiolitis, there were 2 additional guidelines retrieved by this structured gray literature search, a more recently published guideline and an update since the last search was completed. For asthma, the same guidelines were found. Fourth, we chose to limit our search to those in English. This may have unnecessarily narrowed our search, and we may have had a more comprehensive list of international guidelines if we had translated those guidelines in other languages. However, there were only 4 guidelines for asthma that were excluded because of non-English language status. None of the bronchiolitis guidelines found were non-English. This may limit our generalizability of this process for guideline appraisal and comparison with those countries that are non-English speaking, though standardization of the guideline content and quality has been part of a worldwide discussion with the AGREE II tool, which is available and translated into 32 languages.36 Finally, we limited our study to treatment recommendations. Findings may have differed if we had considered other recommendations focused on assessment, emergency referral criteria, presentation, diagnostic testing, follow-up, prophylaxis, prevention, and/or education.

The discrepancies found in agreement between guideline recommendations in common pediatric conditions cared for by a pediatrician or pediatric hospitalist among national and international guidelines is concerning. There is substantial variability in treatment recommendation guidelines among national and international guidelines for asthma and some variation for bronchiolitis. There is variation in guideline development methods across the world. There were over 60 different evidence evaluation and recommendation grading systems in use when last evaluated in 2012, making the interpretation of guidelines more difficult.23,37 Clinical practice guideline panels may benefit from adapting existing evidence synthesis and clinical practice guidelines to their local context rather than from de novo development of evidence synthesis to create a new guideline.8 Clarity and transparency in clinical practice guideline work would improve if there were more collaborative international work in clinical practice guideline development, or, at the least, more fidelity to a standard reporting structure.

Overall κ analysis revealed slight agreement for asthma and fair agreement for bronchiolitis guidelines. This suggests that there is variability in treatment recommendation guidelines among national and international guidelines for asthma and bronchiolitis.

     
  • AAH

    Australian Asthma Handbook

  •  
  • AAP

    American Academy of Pediatrics

  •  
  • ACCP

    American College of Chest Physicians

  •  
  • AGREE II

    Appraisal of Guidelines Research and Evaluation II

  •  
  • CI

    confidence interval

  •  
  • CPS

    Canadian Paediatric Society

  •  
  • CTS

    Canadian Thoracic Society

  •  
  • GINA

    Global Initiative for Asthma

  •  
  • NHLBI

    National Heart, Lung, and Blood Institute

  •  
  • SIGN

    Scottish Intercollegiate Guidelines Network

  •  
  • SNHS

    Spanish National Health System

Dr Bakel conceptualized and designed the study, coordinated and supervised data collection, participated in the analysis of the data, interpreted the results, and drafted the initial manuscript; Dr Hamid helped with the design of the study and data analysis and reviewed and revised the manuscript; Drs Straus, Parkin, and Cohen helped with the design of the study and reviewed and revised the manuscript; Ms Lui and Ewusie participated in data collection and reviewed and revised the manuscript; Mr Mussa participated in data review and correction and reviewed and revised the manuscript; and all authors approved the final manuscript as submitted.

FUNDING: No external funding.

1
Hayward
RS
,
Wilson
MC
,
Tunis
SR
,
Bass
EB
,
Guyatt
G
.
Users’ guides to the medical literature. VIII. How to use clinical practice guidelines. A. Are the recommendations valid? The Evidence-Based Medicine Working Group.
JAMA
.
1995
;
274
(
7
):
570
574
[PubMed]
2
Parikh
K
,
Hall
M
,
Teach
SJ
.
Bronchiolitis management before and after the AAP guidelines.
Pediatrics
.
2014
;
133
(
1
). Available at: www.pediatrics.org/cgi/content/full/133/1/e1
[PubMed]
3
Lenzer
J
,
Hoffman
JR
,
Furberg
CD
,
Ioannidis
JP
;
Guideline Panel Review Working Group
.
Ensuring the integrity of clinical practice guidelines: a tool for protecting patients [published correction appears in BMJ. 2014;348:f1335].
BMJ
.
2013
;
347
:
f5535
[PubMed]
4
Parikh
K
,
Hall
M
,
Mittal
V
, et al
.
Establishing benchmarks for the hospitalized care of children with asthma, bronchiolitis, and pneumonia.
Pediatrics
.
2014
;
134
(
3
):
555
562
[PubMed]
5
Adhyaru
BB
,
Jacobson
TA
.
New cholesterol guidelines for the management of atherosclerotic cardiovascular disease risk: a comparison of the 2013 American College of Cardiology/American Heart Association cholesterol guidelines with the 2014 National Lipid Association recommendations for patient-centered management of dyslipidemia.
Cardiol Clin
.
2015
;
33
(
2
):
181
196
[PubMed]
6
Parisi
P
,
Vanacore
N
,
Belcastro
V
, et al;
“Pediatric Headache Commission” of Società Italiana di Neurologia Pediatrica (SINP)
.
Clinical guidelines in pediatric headache: evaluation of quality using the AGREE II instrument.
J Headache Pain
.
2014
;
15
:
57
[PubMed]
7
Dhaliwal
R
,
Madden
SM
,
Cahill
N
, et al
.
Guidelines, guidelines, guidelines: what are we to do with all of these North American guidelines?
JPEN J Parenter Enteral Nutr
.
2010
;
34
(
6
):
625
643
[PubMed]
8
Tavender
EJ
,
Bosch
M
,
Green
S
, et al
.
Quality and consistency of guidelines for the management of mild traumatic brain injury in the emergency department.
Acad Emerg Med
.
2011
;
18
(
8
):
880
889
[PubMed]
9
Hussain
T
,
Michel
G
,
Shiffman
RN
.
The Yale guideline recommendation corpus: a representative sample of the knowledge content of guidelines.
Int J Med Inform
.
2009
;
78
(
5
):
354
363
[PubMed]
10
McAlister
FA
,
van Diepen
S
,
Padwal
RS
,
Johnson
JA
,
Majumdar
SR
.
How evidence-based are the recommendations in evidence-based guidelines?
PLoS Med
.
2007
;
4
(
8
):
e250
[PubMed]
11
McMurray
J
,
Swedberg
K
.
Treatment of chronic heart failure: a comparison between the major guidelines.
Eur Heart J
.
2006
;
27
(
15
):
1773
1777
[PubMed]
12
Campbell
F
,
Dickinson
HO
,
Cook
JV
,
Beyer
FR
,
Eccles
M
,
Mason
JM
.
Methods underpinning national clinical guidelines for hypertension: describing the evidence shortfall.
BMC Health Serv Res
.
2006
;
6
:
47
[PubMed]
13
Burgers
JS
.
Guideline quality and guideline content: are they related?
Clin Chem
.
2006
;
52
(
1
):
3
4
[PubMed]
14
Burgers
JS
,
Bailey
JV
,
Klazinga
NS
,
Van Der Bij
AK
,
Grol
R
,
Feder
G
;
AGREE Collaboration
.
Inside guidelines: comparative analysis of recommendations and evidence in diabetes guidelines from 13 countries.
Diabetes Care
.
2002
;
25
(
11
):
1933
1939
[PubMed]
15
Brouwers
MC
,
Kho
ME
,
Browman
GP
, et al;
AGREE Next Steps Consortium
.
AGREE II: advancing guideline development, reporting, and evaluation in health care.
Prev Med
.
2010
;
51
(
5
):
421
424
[PubMed]
16
Keren
R
,
Luan
X
,
Localio
R
, et al;
Pediatric Research in Inpatient Settings (PRIS) Network
.
Prioritization of comparative effectiveness research topics in hospital pediatrics.
Arch Pediatr Adolesc Med
.
2012
;
166
(
12
):
1155
1164
[PubMed]
17
National Guideline Clearinghouse
. Fact sheet.
2012
. Available at: www.ahrq.gov/research/findings/factsheets/errors-safety/ngc/national-guideline-clearinghouse.html. Accessed December 11, 2014
18
Guidelines International Network
. About G-I-N.
2010
. Available at: www.g-i-n.net/about-g-i-n. Accessed December 11, 2014
19
Canadian Agency for Drugs and Technologies in Health
. Grey matters: a practical search tool for evidence-based medicine.
2014
. Available at: https://www.cadth.ca/resources/finding-evidence/grey-matters. Accessed December 11, 2014
20
Brassey
J
. Trip database.
2015
. Available at: www.tripdatabase.com. Accessed September 8, 2015
21
Organization for Economic Cooperation and Development
. Members and partners.
2014
. Available at: www.oecd.org/about/membersandpartners/. Accessed March 16, 2015
22
Andrews
JC
,
Schünemann
HJ
,
Oxman
AD
, et al
.
GRADE guidelines: 15. Going from evidence to recommendation-determinants of a recommendation’s direction and strength.
J Clin Epidemiol
.
2013
;
66
(
7
):
726
735
[PubMed]
23
American Academy of Pediatrics Steering Committee on Quality Improvement and Management
.
Classifying recommendations for clinical practice guidelines.
Pediatrics
.
2004
;
114
(
3
):
874
877
[PubMed]
24
Jadad
AR
,
Moher
M
,
Browman
GP
, et al
.
Systematic reviews and meta-analyses on treatment of asthma: critical evaluation.
BMJ
.
2000
;
320
(
7234
):
537
540
[PubMed]
25
Cohen
J
.
A coefficient of agreement for nominal scales.
Educ Psychol Meas
.
1960
;
20
(
1
):
37
46
26
R Core Team
.
R: A Language and Environment for Statistical Computing
[computer program].
Vienna, Austria
:
R Foundation for Statistical Computing
;
2015
27
Landis
JR
,
Koch
GG
.
The measurement of observer agreement for categorical data.
Biometrics
.
1977
;
33
(
1
):
159
174
[PubMed]
28
Feuerstein
JD
,
Akbari
M
,
Gifford
AE
, et al
.
Systematic analysis underlying the quality of the scientific evidence and conflicts of interest in interventional medicine subspecialty guidelines.
Mayo Clin Proc
.
2014
;
89
(
1
):
16
24
[PubMed]
29
Matthys
J
,
De Meyere
M
,
van Driel
ML
,
De Sutter
A
.
Differences among international pharyngitis guidelines: not just academic.
Ann Fam Med
.
2007
;
5
(
5
):
436
443
[PubMed]
30
Reddy
AP
,
Gupta
MR
.
Management of asthma: the current US and European guidelines.
Adv Exp Med Biol
.
2014
;
795
:
81
103
[PubMed]
31
Guillén
Ú
,
Weiss
EM
,
Munson
D
, et al
.
Guidelines for the management of extremely premature deliveries: a systematic review.
Pediatrics
.
2015
;
136
(
2
):
343
350
[PubMed]
32
Koh
C
,
Zhao
X
,
Samala
N
,
Sakiani
S
,
Liang
TJ
,
Talwalkar
JA
.
AASLD clinical practice guidelines: a critical review of scientific evidence and evolving recommendations.
Hepatology
.
2013
;
58
(
6
):
2142
2152
[PubMed]
33
Hester
G
,
Nelson
K
,
Mahant
S
,
Eresuma
E
,
Keren
R
,
Srivastava
R
.
Methodological quality of national guidelines for pediatric inpatient conditions.
J Hosp Med
.
2014
;
9
(
6
):
384
390
34
Woods
CR
.
AAP clinical guidelines: ongoing process improvements.
Pediatrics
.
2013
;
131
(
4
):
794
795
[PubMed]
35
Viera
AJ
,
Garrett
JM
.
Understanding interobserver agreement: the kappa statistic.
Fam Med
.
2005
;
37
(
5
):
360
363
[PubMed]
36
AGREE
. Advancing the science of practice guidelines. Available at: www.agreetrust.org. Accessed November 14, 2015
37
Bai
ASV
,
Bak
G
,
Wells
G
.
Quality Assessment Tools Project Report
.
Ottawa, Canada
:
Canadian Agency for Drugs and Technologies in Health
;
2012

Competing Interests

POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

FINANCIAL DISCLOSURE: Dr Straus is funded by a Tier 1 Canada Research Chair in Knowledge Translation; the other authors have indicated they have no financial relationships relevant to this article to disclose.

Supplementary data