The "Utah Review" of Hormonal Treatments for Gender-Dysphoric Minors: A Methodological Appraisal

Systematic reviews are a cornerstone of evidence-based decision-making because they provide the most reliable evidence regarding the benefits and harms of medical treatments. The Utah Legislature explicitly required a “systematic evidence review” of the effects of puberty blockers and cross-sex hormones as the basis for deciding whether to lift the moratorium on their use in minors. Given the centrality of this “systematic review” requirement, it is important to determine whether the Utah Review meets established methodological standards for a systematic review.

The research protocol provides an important basis for assessing any review’s adherence to a systematic review methodology. The protocol describes the review’s objectives, eligibility criteria, and analytic methods—all of which must be set out before the analysis begins. Best practices call for protocols to be shared via registries such as PROSPERO or the Open Science Framework. When that is not feasible, the protocol should be published with the final review, with any deviations transparently disclosed and justified.

While the Utah Review references the existence of prespecified internal protocols (p. 12), it does not make them available. As a result, evaluating the research process requires analysis of multiple sources, including the S.B. 16 bill that commissioned the Utah Review, the University of Utah DRRC research proposals, the interim results presentation, and the Evidence Review and Recommendation Report.

The need for such reconstruction, rather than straightforward verification against a prespecified protocol, is itself a methodological concern because it limits the ability to independently assess the rigor of the review. Accordingly, we present a constrained analysis based on the available materials, highlighting what appear to be significant deviations from planned methods and standard research practices.

Although specific approaches to systematic reviews may vary, systematic reviews include the following core steps:

1. Determine the research questions and eligibility criteria

2. Search for evidence and select studies
3. Abstract data and assess risk of bias in eligible studies
4. Create a formal evidence synthesis for each outcome
5. Assess the quality/certainty of the evidence and present evidence-based conclusions

The rest of this section provides an assessment of the extent to which the Utah Review’s methodological process adheres to systematic review methodology. Subsequent sections elaborate on each analysis in detail.

Research questions, eligibility criteria, and study selection (Steps 1, 2)

The systematic review process begins by articulating key research questions in PICO (Population, Intervention, Comparator, Outcome) format. While the Utah Review articulated the definition of the “population” (patients <18 years with gender dysphoria and related conditions) and the “interventions” (hormonal interventions used in the context of treating gender dysphoria), the definitions of “comparator” and “outcome” appear to have undergone significant revisions throughout the process—often without a methodologically appropriate justification, introducing bias.

Initially, the Utah Evidence Review planned to allow all study types, regardless of whether the study used a comparative design or which comparator was used, and it also intended to treat all reported outcomes as outcomes of interest (p. 12). This broad strategy resulted in the identification of an infeasibly large number of studies. To narrow the analytic study set, the researchers created a post-hoc category of “high priority” study types and outcomes and then analyzed only studies with outcomes that met the new definition of “high priority.”

During this post-hoc revision, the researchers made the decision to not consider or to explicitly remove the following outcomes from their “high priority” list:

• Fertility. The accompanying Recommendation Report states that “fertility” was not considered a high-priority outcome because “infertility is a known risk of CSHT [cross-sex hormone therapy] and was not an outcome of focus in the DRRC systematic review” (Recommendation Report, 4). This is a problematic decision given that the Legislature explicitly requested consideration of the “harms of interrupting natural puberty” and fertility counseling is a core component of pediatric gender transition protocols precisely because loss of fertility is a possible consequence.
• Desistance. The Utah Legislature explicitly requested that the Utah Review address the “rates of desistance and time to desistance.” The University of Utah DRRC acknowledged that desistance and related concepts were “pointedly of interest to the legislature,” and included the question of short- and long-term rates of discontinuation in its contract with Utah DHHS and as a research objective in the Review itself. Despite this, the Review also informs readers that the researchers chose not to consider them “high priority” outcomes (Evidence Review, p. 83).
• Regret. Initially, it appears that “regret” was a “high priority outcome” (see Figure 2 below). However, in the final Evidence Review, “regret” was no longer listed as such. No explanation is available for this change; instead, the Evidence Review explicitly states that although “regret” was “pointedly of interest to the legislature,” it was “not among our high-priority outcome categories for this review” (Evidence Review, p. 83).

Post-hoc exclusions of these patient-important outcomes, some of which were identified as “pointedly of interest” to the Review’s commissioning body, are inherently problematic. Furthermore, since “high-priority” outcomes were used to separate studies into “relevant” and “not relevant,” omitting outcomes pertaining to fertility, regret, or detransition/desistance from the high-priority list meant that studies focused on those outcomes were removed from the analytic set. As a result, the Evidence Review’s conclusions systematically overlooked the very domains where harms are likely to be observed.

Figure 2. “High-Priority” Studies, Presentation to the Utah DHHS Interim Committee. Hofmann, June 2024, SB16 (2023) Transgender Medical Treatments and Procedures Amendments Progress Report (1:19:54-1:26:57)

Abstracting data and risk of bias assessment (Step 3)

All studies that meet eligibility criteria typically undergo risk of bias assessment, even if only a subset is later included in the subsequent step of evidence synthesis. While the Utah researchers identified a total of 277 eligible studies, fewer than 40% (101/277) underwent data extraction and risk of bias assessment (p. 34). The removal of over 60% of eligible studies from the analysis, apparently due to “insufficient resources” (Recommendation Report, p. 4), introduced a critical risk of bias into the Evidence Review’s conclusions.

In addition, a number of the studies subjected to risk of bias analysis received inflated quality ratings, in part because uncontrolled studies with subgroup analyses (for example, males and females) were mistakenly treated as though they were comparative studies. For example, Chen et al. (2023) was given a near-perfect study quality score (8/9 on the Newcastle-Ottawa Scale) with the justification that the study’s “non-exposed cohort” was “drawn from the same community as the exposed cohort” (p. 556). However, Chen et al. lacked a comparator group—as the study's authors openly acknowledged (Chen et al., 2023, p. 249).

Evidence synthesis and quality/certainty of evidence assessment (Steps 4, 5)

An evidence synthesis that assesses the body of evidence for quality (also known as certainty) is a hallmark of any systematic review. The researchers explicitly acknowledge omitting this analysis: “We were not contracted to include a synthesis of the evidence that we found: only to assess ROB [risk of bias] and provide evidence tables summarizing safety and efficacy” (p. 90).

For clarity, risk of bias in Step 3 is assessed at the individual study level and addresses whether features of a study’s design, conduct, or analysis may have biased the estimated intervention effect. By contrast, synthesis and assessment of the body of evidence for quality/certainty are conducted for each specific outcome across studies. This can be done using GRADE (Grading of Recommendations Assessment, Development and Evaluation) or through narrative synthesis methods, as in the University of York systematic reviews.

These steps are important because they move the analysis beyond individual studies to assess what the totality of the evidence shows about a specific outcome, and how much confidence that conclusion warrants. Whether conducted using GRADE or narrative synthesis, this process requires reviewers to weigh factors such as study design and risk of bias, the consistency (or inconsistency) of findings across studies, heterogeneity, and the magnitude of observed effects to judge how trustworthy the body of evidence is.

Because the Utah Review did not undertake these essential steps, it does not meet the criteria for a systematic review. A risk of bias assessment alone is not sufficient; reviewers must also synthesize the evidence for each outcome and assess the certainty of the body of evidence.

Presentation of the Utah Review as a “systematic review”

It remains unclear why the University of Utah DRRC team, described as “experts in the methodology of evidence synthesis” by DHHS (Recommendation Report, p. 2), responded to a mandate to conduct a “systematic evidence review” with an analysis plan that omitted a key step for such a review—a formal evidence synthesis that assesses the quality/certainty of evidence.

Part of the problem may stem from the fact that the work was guided by a pre-existing evidence review contract between Utah DHHS and the University of Utah College of Pharmacy’s DRRC department. The purpose of the ongoing multi-year contract is to support routine Utah Medicaid pharmacy decision-making, such as identifying prior authorization criteria (First Contract Amendment). To support the Utah Legislature's request for a systematic review related to pediatric gender dysphoria, the contract was updated with Attachment D, “Gender Dysphoria Report Proposal,” submitted by the University of Utah DRRC team (First Contract Amendment, Attachment D).

While the contract contains general language stating that the Utah DRRC team would “perform systematic reviews of the evidence,” nothing in the main contract itself or Attachment D commits the researchers to adhering to an explicit systematic review methodology, such as performing an assessment of the body of evidence for quality/certainty using GRADE or similar evidence synthesis methods.

Although Attachment D states that the research would assess the “quality … of evidence” (First Contract Amendment, p. 6), the DRRC team equated this analysis with “risk-of-bias [ROB]” assessment of individual studies—a distinct analytic exercise. This conflation suggests either a misunderstanding of research methodology or a deliberate departure from established methods. In either case, the absence of an explicit assessment of the body of evidence for quality/certainty—an essential component of any systematic review—indicates that the Utah Review does not meet the criteria for a systematic review of evidence.

While the Utah Review is demonstrably not a systematic review, multiple stakeholder groups have described the Utah Review as a “systematic review” and emphasized that designation as a marker of methodological rigor:

• The University of Utah DRRC research team described their analysis as a “systematic review” in the Review itself (p. 899). 
• The University of Utah DRRC presentation of the Review’s results at the American Academy of Pediatrics’ 2024 national conference described the analysis as a “systematic review.”
• Utah DHHS described the work as a “systematic review” during the interim results’ presentation (DHHS presentation, slides 8–9) and also in its final Recommendation Report (Recommendation Report, p. 5, 6, 10).
• A congressional comment letter signed by 106 Members of Congress described the Utah Review as a “systematic review.”
• A letter from the American Public Health Association to the U.S. Department of Health and Human Services, signed by 123 public health and health policy deans, chairs, and scholars, described the Utah Review as a “comprehensive systematic review.”

While the Utah Review does not contain any analyses that represent “systematic reviews,” it does present literature reviews. Since literature reviews (also known as "narrative reviews") do not use systematic, reproducible methods to identify, appraise, and synthesize evidence, they are vulnerable to selection and interpretive bias and less reliable for drawing conclusions about treatment effects.

The following sections examine each of the Utah Review’s analyses to assess whether the methods used are rigorous and whether the conclusions are trustworthy.

3. Identification of Pharmacological Agents

The identification of pharmacological agents and their Food and Drug Administration (FDA) licensing status, the first analysis undertaken by the Utah Review, served two purposes. First, the analysis fulfilled the requirements of the Utah Legislature by identifying the hormonal interventions used in the treatment of pediatric gender dysphoria. Second, the names of the identified drugs became the search terms for the literature search that formed the evidentiary basis for the rest of the analyses.

The University of Utah DRRC team used standard databases including Micromedex, UpToDate, and the FDA Orange Book to “identify a comprehensive list of all drug product hormones and hormonally active agents that are administered to pediatric TGNB [transgender and nonbinary] patients in the United States” (p. 6). The researchers identified 66 such drugs and reported that none was FDA-approved for that indication.

However, the analysis went beyond stating these findings as facts, and introduced a narrative supporting off-label use of hormonal agents in gender-dysphoric youth:

• Justifying off-label drug use on the basis of weak evidence. In describing the high prevalence of off-label prescribing in pediatric practice (“as high as 38.1% of prescriptions, and as many as 78.9% of children”) (p. 6), the Review implied that the principal obstacle to obtaining an on-label indication for hormonal interventions for minors is the added cost of obtaining regulatory approval. The Review did not mention alternative explanations, such as the possibility that the use of these interventions in pediatric populations may be unable to meet the FDA evidentiary threshold.
• Drawing a false equivalence to routine pediatric off-label prescribing. The researchers invoked the common pediatric practice of prescribing antibiotics for infections to suggest that endocrine interventions for gender dysphoria in youth represent a comparable clinical practice. However, they failed to address the key differences between these contexts. Antibiotics are intended for short-term use, have been extensively studied in adults, and are used to treat conditions with a well-characterized disease course, providing a strong evidentiary basis for pediatric extrapolation. By contrast, endocrine interventions for gender dysphoria have not demonstrated clear benefit in adult populations, and the combination of puberty blockers with cross-sex hormones has bypassed even preclinical and early-phase clinical trial evaluation. Moreover, unlike short courses of antibiotic therapy, masculinizing and feminizing hormones are typically intended for lifelong use and will have irreversible lifelong effects even if discontinued.
• Framing off-label prescribing as intrinsically safe. In discussing safety considerations, the researchers did not mention multiple studies demonstrating that off-label use in pediatrics is associated with worse health outcomes and a significantly increased risk of adverse reactions and events (risk/odds ratios of 1.67–2.25). Instead, they cited “one study [that] showed no differences in the risk of adverse events” associated with off-label prescribing (p. 6). Further, the fact that some interventions may be offered based on weak evidence does not necessarily justify additional off-label uses; instead, it highlights a broader problem requiring closer scrutiny.

Notwithstanding these limitations, the review of agents provides a useful contribution to an understanding of the clinical realities of treating pediatric gender dysphoria.

4. Evidence Review of Clinical Studies

The analysis of primary clinical studies, frequently cited for including “more than 28,056 pediatric patients” (p. 44), constitutes the bulk of the Evidence Review. On this basis, it has been consistently characterized as a comprehensive systematic evidence review by University of Utah researchers and Utah DHHS (Interim DHHS presentation, slide 8; Recommendation Report, p.5), as well as by some users of the report, including policymakers.

Systematic reviews sit at the top of the evidence pyramid because they examine the totality of available evidence in a systematic and reproducible manner. A credible systematic review addresses a predefined question, applies explicit study selection criteria, conducts a systematic search across multiple sources, synthesizes characteristics of the studies and findings across all studies at the outcome level, and assesses the quality/certainty of the evidence. These features distinguish systematic reviews from literature (narrative) reviews, which are not reproducible, and from scoping reviews, which may be reproducible but do not evaluate treatment effects.

We have applied two well-recognized systematic review assessment tools to assess the quality of the Evidence Review’s analysis of primary clinical studies: ROBIS (Risk Of Bias In Systematic reviews) and AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews).

These analyses yielded ratings of “high risk of bias” and “critically low” confidence respectively—the lowest ratings possible.

• ROBIS. ROBIS is a widely used, structured tool for appraising systematic reviews for “risk of bias”—the degree to which the review process (how it was designed, conducted, or interpreted) may have skewed conclusions. According to the ROBIS assessment of the Utah analysis of primary studies, serious methodological concerns were identified across all four ROBIS domains, yielding the overall rating of “high risk of bias”—the lowest possible rating (Table 1, below). For the full ROBIS assessment see Supplement 1.

Table 1. ROBIS Assessment of the Evidence Review of Clinical Studies: Summary

• AMSTAR 2. AMSTAR 2 is a methodological tool designed to assess the overall quality of systematic reviews of healthcare interventions. Applying AMSTAR 2 criteria to the Utah analysis of primary clinical studies reveals problems in four “critical” domains, including the absence of a pre-registered protocol or prospectively defined methods (critical domain 2), inadequate risk of bias assessment in individual studies (critical domain 9), failure to incorporate risk of bias assessments into the review’s conclusions (critical domain 13), and failure to evaluate publication bias (critical domain 15). The overall rating of “critically low” confidence places the Utah analysis of primary clinical studies in the lowest possible quality category. For the AMSTAR 2 assessment see Supplement 2.

Taken together, ROBIS and AMSTAR 2 point to the same conclusion: the findings of the Evidence Review’s analysis of primary clinical studies are not trustworthy and the analysis should not be relied upon to provide an accurate and comprehensive summary of the available evidence.

Some of the most significant deficiencies are:

• Lack of a prespecified protocol. A clearly articulated research protocol is essential for the transparency and reproducibility of systematic reviews. While the researchers assert that some prespecified “internal protocols” existed for some of the steps they followed (p. 12), the protocol is not presented anywhere in the Evidence Review. Further, the research process underwent significant post-hoc revisions without a methodologically acceptable rationale.
• Unfocused eligibility and inclusion criteria. All study types reporting any outcomes related to hormonal treatments for minors with gender dysphoria were considered “eligible” for inclusion in the analysis. This broad strategy yielded 230 studies (p. 34), which is unusually large for this type of analysis. For context, McMaster University systematic evidence reviews of the same topic conducted around the same time, which had a higher maximum age of 25, judged only 10 puberty blocker and 24 cross-sex hormone studies as eligible for inclusion. Because the University of Utah researchers were unable to analyze this large study set, arbitrary post hoc decisions were taken to limit the number to a more manageable size, thereby introducing selection bias.

• Exclusion of important outcomes. The researchers reduced the list of 230 eligible studies to a smaller set of 90 "relevant" studies by only analyzing the studies that contained “high priority” outcomes. The definition of “high priority” appears to have been constructed post hoc, and did not include important outcomes, such as fertility, regret, desistance/detransition, sexual function, and mortality. Consequently, any eligible studies focusing on these outcomes were filtered out as "not relevant" and did not inform the analysis or its conclusions.

• Failure to analyze over 60% of eligible studies. Only 90 of the 230 eligible studies were subjected to subsequent analysis. The remaining 140 eligible studies (60%) did not undergo data extraction and risk of bias assessments, and were relegated to bibliography-only, which is a major deviation from standard research methods.

  The final set of 90 “relevant” studies fails to form a cohesive analytic framework and instead represents a heterogeneous mix of populations, interventions, outcomes, and study designs. Below are examples of studies that made it to the final “relevant” study list:

  • A study of insurance coverage in patients treated at a single-site clinic.
  • A study of how the brains of testosterone-treated adolescent females perceive familiar versus unfamiliar voices.
  • A study comparing the use of different doses of the same pharmacological puberty-suppression agent.

   

  At the same time, studies of rates of discontinuation of hormonal treatment, fertility and sexual function outcomes, regret, and mortality were deemed not “relevant” and had no opportunity to inform the conclusions of this analysis.

• Inaccurate study assessments and inflated quality ratings. Even after reducing the number of included studies, the researchers were still left with 90 studies—a large number—to assess for risk of bias. Best practices recommend that at least two researchers should be involved in extracting data from the studies and conducting risk of bias assessments. The University of Utah DRRC team deviated from this best practice and instead relied on a single researcher, which rendered the analysis vulnerable to errors, with an apparent bias toward inflated study quality ratings. The Review’s assessment of two well-known studies illustrates this:

  • Chen et al., 2023: The University of Utah DRRC researchers assigned Chen et al., 2023 a near-perfect score of 8/9 on the Newcastle-Ottawa Scale (NOS), despite the study’s well-documented limitations. The researchers treated this study as having “exposed” and “non-exposed” cohorts, stating that the non-exposed cohort was “drawn from the same community as the exposed cohort” (p. 556), suggesting a robust study design. However, Chen et al. explicitly acknowledged that the study lacked a non-exposed comparison group: “… our study lacked a comparison group, which limits our ability to establish causality” (Chen et al., 2023, p. 249).

    The assessment also overlooked that Chen et al. failed to report most prespecified outcomes, had approximately 30% attrition at 24 months that was unexplained (aside from noting that two youths died by suicide during treatment), and did not adequately control for important confounders. An independent assessment of this study assigned it five stars on the NOS, which suggests moderate quality, while the more rigorous ROBINS-I assessment from the same source deemed it at “critical risk of bias”—the lowest rating possible.

  • Tordoff et al., 2022: The researchers assessed Tordoff et al. as moderate quality, assigning it an NOS score of 6/9, despite the study’s serious methodological limitations (p. 570). Although the researchers assessed the study as though it had an untreated comparator group, the majority of participants in that group either crossed over to the treatment arm over the 12-month duration of the study, or dropped out from the gender clinic and the study entirely, leaving only seven untreated subjects at the end of follow-up, compared with the original 97 (Tordoff et al., 2022, Supplemental online content, eTable 2). Further, since all patients were eligible for hormonal treatment unless they had severe mental health problems that prevented consent, youth who remained untreated despite continuing to attend the gender clinic may have differed systematically at baseline from those who received treatment, further compromising comparability.

    Moreover, nearly 40% of participants dropped out (39/104) at 12-month follow-up, which itself was too short to meaningfully assess long-term outcomes. Of note, the York systematic review, which evaluated the same study using the same tool (NOS), gave it a much lower score of 3.5, classifying it as a low-quality study.

• Missing evidence synthesis assessing quality/certainty of evidence. A formal evidence synthesis, which includes an assessment of the evidence for quality/certainty, is a cornerstone of a trustworthy systematic review. The University of Utah DRRC researchers did not synthesize results at the outcome level across studies and did not assess the body of evidence for certainty using either the GRADE assessment or a narrative synthesis (as was done in the U.K. York systematic reviews).

  The researchers explain this omission by stating that they “were not contracted to include a synthesis of the evidence that we found: only to assess risk of bias and provide evidence tables summarizing safety and efficacy findings” (p. 90). It is unclear why the DRRC research team advanced a methodology for a “systematic review” that omits one of the most critical aspects of a systematic review—a formal evidence synthesis. As a result, readers have no basis for determining whether their findings are based on robust comparative data or are derived from low-quality observational evidence and small samples with high attrition.

  For example, they assert conclusions such as “rates of depression and suicidal thoughts/self-harm tended to be lower among hormonally treated transgender youth compared to untreated transgender individuals” (p. 77) without specifying 1) which treatments were assessed; 2) how many studies contributed to this conclusion; 3) how many participants were included; 4) which specific outcome measures were assessed; or 5) the magnitude of effect sizes.

• Deviations from accepted research processes to arrive at conclusions. The University of Utah DRRC team stated that, “[a]fter having spent many months searching for, reading, and evaluating the available literature, it was impossible for us to avoid drawing some high-level conclusions” (p. 90). The researchers’ conclusion about the use of puberty blockers and cross-sex hormones was that “the consensus of the evidence supports that the treatments are effective” in improving mental health and that the “evidence also supports that the treatments are safe in terms of changes to bone density, cardiovascular risk factors, metabolic changes, and cancer” (p. 90).

  However, “consensus of the evidence” is not a recognized term in evidence-based medicine. Acceptable terminology refers to the “body of evidence” or the “totality of evidence” and such conclusions require adherence to systematic review methodology, which assesses the quality/certainty of the evidence.

• High number of errors. The analysis contains multiple errors and irregularities. While the text claims that “there were N=134 primary clinical studies reporting findings in TGNB [transgender and nonbinary] populations all over the world” (p. 44), the relevant table and appendix include 230 studies (pp. 489–538), and the same number—230—is referenced in the conclusion (p. 90). At several points, the researchers conflate two different studies by Klaver et al. In Nahata et al. (2017), anxiety counts were misreported (p. 659). In van de Grift et al. (2020), follow-up age and several outcome counts were inaccurately recorded (p. 726–729). Spot-checking suggests that such errors are not isolated. The apparent high prevalence of this type of error is likely due, in part, to the researchers’ deviation from the established practice of having two researchers verify extracted data.

• Incoherent presentation of study detail. Over 90% of the 1,051-page document consists of data tables and other material normally considered supplementary. Absent a proper evidence synthesis, much of the information about the evidence assessed must be gathered from these data tables. Unfortunately, this material is challenging to interpret due to poor organization and a high number of redactions of basic study information, purportedly to “protect” the identities of patients and clinical centers (see Figure 3).

  It is unclear what the redactions achieved beyond making parts of the report incomprehensible, since the information the authors sought to conceal was basic study information already available in the underlying studies, all of which are in the public domain.

Figure 3. Example of redactions (pp. 49-50).

As our analysis demonstrates, despite being presented as a systematic review, the University of Utah DRRC analysis of primary clinical studies omitted a fundamental requirement—namely, a formal evidence synthesis assessing the evidence for quality/certainty. This reduces the analysis, at best, to a “literature review” rather than a “systematic review.” Even as a “literature review,” the analysis suffers from profound limitations. As such, the conclusions of the Utah Review on primary clinical studies cannot be considered a credible basis for evidence-based decision-making.

5. Overview of Systematic Reviews

Overviews of existing systematic reviews (also known as “umbrella reviews”) typically follow the same methodology as systematic reviews of clinical studies; the key difference is that the unit of analysis is a systematic review rather than a primary clinical study. Otherwise, similar systematic methods are expected at every stage of the research process. To the extent that umbrella reviews are used to draw conclusions about the effects of interventions, they must assess the evidence for certainty for each outcome of interest through formal evidence synthesis.

The University of Utah DRRC research team analyzed seven systematic reviews. As with the systematic review of clinical studies, the most serious deficiency in this analysis is the absence of an evidence synthesis and an assessment of the quality/certainty of the evidence. We assessed the Utah Review’s analysis of systematic reviews using the PRIOR (Preferred Reporting Items for Overviews of Reviews) checklist for umbrella reviews. Several key limitations are outlined below:

• Inadequate and outdated search strategy. The researchers overlooked major databases and systematic review repositories—including the Cochrane Database of Systematic Reviews (CDSR), PROSPERO, CINAHL, and Epistemonikos. They also failed to search relevant gray literature, such as systematic reviews posted on public health authorities’ websites, including the U.K. NICE evidence reviews (2020) on puberty blockers and cross-sex hormones.

  In addition, the researchers did not update their search after the publication of the seminal Cass Review, which relied on the University of York systematic reviews on puberty blockers and cross-sex hormones. These highly relevant systematic reviews were released nearly a year before the public release of the Utah Review. As a result, the Utah analysis failed to incorporate a body of systematic review evidence that proved foundational to U.K. policy restrictions on these hormonal interventions for minors with gender dysphoria.

• Subjective systematic review selection. The research team used a broad definition of “systematic review,” treating any evidence review that describes itself as “systematic” as such, regardless of the methods used. The research team identified 38 systematic reviews, seven of which were designated “high priority” for evaluation. However, the inclusion criteria appear to be inconsistently applied. For example, Mahfouda et al. (2019)—a literature review which did not follow a systematic review methodology and never claimed to be a systematic review—was included. Its inclusion may be due to the fact that it is commonly cited in support of pediatric gender transition, but it nevertheless failed to meet the Evidence Review’s eligibility criteria.

• Evidence of bias in reviews’ assessments. The seven systematic reviews identified as relevant were subjected to AMSTAR 2 analysis (p. 43). A review of the quality ratings suggests evidence of researcher bias. For example, the systematic review by Ludvigsson et al., 2023, which originated from the work of Sweden’s SBU and underpinned that country’s 2022 restrictive guidelines, was assessed as a low quality review and an “outlier in terms of several key methods” (p. 43, 63). The University of Utah researchers asserted that because the Swedish researchers excluded high-risk-of-bias studies from the evidence synthesis, they committed a major methodological violation. This is inaccurate. Excluding high risk of bias studies from a formal evidence synthesis is an acceptable approach and is recommended in the Cochrane Handbook.

  Furthermore, the discussion section focused almost exclusively on criticizing the Swedish review and questioning its conclusions (p. 42, 63). At the same time, systematic reviews whose conclusions are supportive of the practice of pediatric transitions—including those that received similarly low ratings—were not mentioned in the discussion.

  Of note, like all research, the Swedish systematic review has limitations, but it was judged to be among the field’s higher-quality systematic reviews by a recent umbrella review appraisal.

• Missing evidence synthesis assessing quality/certainty. As with the analysis of primary clinical studies, the University of Utah’s analysis of systematic reviews lacks a key component of a systematic review—an evidence synthesis which summarizes what is known about each outcome of interest, and how certain we can be in that knowledge.

Given the profound limitations of this analysis, culminating in the omission of a formal evidence synthesis assessing the body of evidence for quality/certainty, the Utah Review’s analysis of systematic reviews cannot be considered a systematic evidence review. Consequently, it cannot be used as the basis for evidence-based decision-making.

6. Long-term Outcomes

One of the key requirements of the Utah Legislature was that the evidence review assess long-term outcomes of hormonal interventions. When the researchers presented the analysis results in February 2024, it was noted that most included studies followed patients for only 1–2 years after hormone initiation and therefore did not provide meaningful long-term data (p. 899). The University of Utah DRRC research team was asked to revisit long-term outcomes. Subsequently, the researchers issued a new research proposal. This analysis of 17 “long-term” studies became a separate research output comprising Part II of the Evidence Review (p. 898).

With respect to benefits, the researchers noted that some studies reported psychological improvements associated with hormonal treatment, including reductions in anxiety, depression, and social distress, as well as improvements in other measures of mental health and functioning, while other studies found no significant improvements.

With respect to harms, the researchers observed that some studies reported statistically elevated risks of cardiovascular mortality associated with ethinyl estradiol use, increased incidence of certain benign brain tumors, and higher suicide and all-cause mortality rates compared with the general population, whereas other studies did not detect statistically significant increases in the outcomes assessed.

The long-term outcomes analysis was based on the same study search and followed the same methodological approach as the Part I analysis of the primary clinical studies (with the exception of adding one more “high priority” outcome of “mortality”); it is therefore subject to the limitations described in the earlier sections. Below, we highlight the additional irregularities in the research process that uniquely affected the long-term outcomes analysis:

• Inconsistent treatment of studies of adults. The researchers did not conduct a separate literature search to support the long-term outcomes analysis, asserting that the prior search was “comprehensive enough” (p. 899). Instead, they merely re-screened previously identified evidence, looking for studies of patients who were at least five years post-treatment initiation. However, since the original search was explicitly governed by pediatric search terms such as “Pediatrics,” “Child,” “Minors,” “Adolescent,” and “Puberty” (see Figure 4), this strategy introduced serious methodological inconsistencies in how the research engaged with studies of hormonally treated adults.

   

  Figure 4. Necessary search criteria for studies of long-term harms.

Because the search was limited to pediatric-specific search terms, adult-only outcome studies were missed. However, whenever adult outcomes happened to be reported alongside pediatric outcomes, they were allowed to enter the analytic set—even when such studies had very few pediatric patients (e.g., de Blok et al., 2021). As a result, the selection of long-term outcome studies was driven more by happenstance than by any cogent methodological principle.

Not only did this introduce incoherence into the evidence base—with some, but not other, adult studies informing the long-term analysis—but this search strategy also excluded some of the most robust evidence of long-term outcomes of hormonal treatments, which come from studies of hormonally treated adults (for example, Bränström and Pachankis (2019), Dhejne et al. (2011), and Jackson et al. (2023)). Notably, the omitted studies consistently report elevated rates of morbidity and mortality among hormonally treated individuals.

• Lack of adherence to the stated inclusion criteria. In re-screening the original pediatrics-focused study pool for the long-term outcome analysis, the researchers looked for studies reporting outcomes at least five years post-transition-initiation. Yet some studies that met this criterion were excluded for unclear reasons. For example, the researchers excluded Klaver et al. (2020)—a seven-year follow-up study of adolescents. De Blok et al. (2019) is also missing, despite meeting the stated eligibility criteria. Of note, de Blok et al. reported a higher incidence of breast cancer among males treated with estrogen than in the general male population. These exclusions raise concerns that study selection was not conducted in a systematic and reproducible manner.

• Results dominated by a single patient cohort with questionable eligibility. Over 80% of the 10,147 subjects with long-term outcomes in the Utah Review (p. 904) came from a single study of the Amsterdam patient cohort (Wiepjes et al., 2020). That study includes patients at all stages of transition, including some who had not received hormones. It is also unclear whether the study met the “long-term” eligibility threshold of ≥5 years after treatment initiation, because it does not report the average duration of hormone exposure or specify how many participants had been on hormones for at least five years.

• Underestimated risk of bias due to unrecognized confounding and co-interventions. Long-term hormone outcome studies are especially vulnerable to confounding: patients receiving cross-sex hormones often differ from the general population in baseline mental health and socioeconomic status and frequently receive co-interventions, such as psychotherapy or surgery. This makes it difficult to isolate the independent effects of hormones. These limitations do not appear to have been adequately recognized in the Utah Review’s risk of bias appraisal of individual studies.

  The Utah Review’s assessment of the “Dutch Protocol” study (de Vries et al., 2014), which launched pediatric gender transition worldwide, illustrates these problems. The Review awarded that study 6 out of 9 points on the NOS, thereby characterizing it as of moderate quality with respect to long-term hormonal outcomes. But the de Vries et al. study cannot provide reliable evidence on the long-term effects of hormones because it never reported on the outcomes of cross-sex hormones, and all participants underwent surgery before the final assessment. The NICE systematic review of cross-sex hormones conducted in 2020 recognized this and excluded the study from the analysis, stating, “all participants had surgery after gender-affirming hormones. Unable to determine whether changes were due to hormones or surgery” (NICE, p. 72). This is a critical limitation since the Utah Review was focused on evaluating hormones, not surgery.

• Failure to integrate long-term findings into overall conclusions. Even though much of the relevant long-term evidence was overlooked, the 17 analyzed studies did identify signals of long-term harm associated with hormonal interventions, particularly elevated mortality. However, these conclusions from Part II were not incorporated into the main Evidence Review’s conclusions, which continued to state that “the evidence … supports that the treatments are safe in terms of changes to bone density, cardiovascular risk factors, metabolic changes, and cancer” (p. 90).
• Absence of assessment of evidence for certainty. As with other analyses, no formal evidence synthesis to assess the evidence for certainty was performed.

Overall, the Utah Review’s long-term outcomes analysis is insufficiently rigorous to support conclusions about the long-term safety or benefits of endocrine interventions. The failure to identify key long-term studies and to integrate signals of harm, such as elevated mortality, into the Utah Review’s main conclusions further undermines its value for evidence-based decision-making.
 

7. Summary of Clinical Practice Guidelines

Synthesizing multiple clinical practice guidelines poses recognized methodological challenges: recommendations are often numerous and qualitative, terminology and scope vary, interventions may not align, and evidence-grading systems often differ, with identical labels reflecting different standards. These complexities require explicit, systematic methods for comparison, synthesis, and transparent reporting (Johnston et al., 2019)—typically through a systematic or scoping review.

The application of the Johnston et al. (2019) framework for assessing the quality of the Utah Review’s synthesis of clinical guidelines indicates that the University of Utah DRRC analysis failed to meet the majority of the criteria, including lack of a prespecified protocol; inadequate search; failure to assess guideline quality; and failure to identify gaps, inconsistencies, and trends across guidelines.

Below, we provide a high-level summary of the key limitations of the guideline analysis.

• Unusual and subjective definition of guidelines for inclusion. The Utah Review adopted an unusual criterion for inclusion of guidelines, asserting that any “guidance from a recognized medical authority” was worthy of inclusion as long as a systematic review was conducted “for at least 1 part of the guideline,” even if that aspect of the guideline was not relevant to the topic of adolescent hormonal treatments. For guidance documents where adolescent recommendations did not include a systematic review, it was sufficient to merely “cite and discuss published literature” (p. 17-18).

  This criterion has a strong subjective component (i.e., “recognized medical authority”), making the inclusion criteria vulnerable to bias. It is also methodologically difficult to justify, since the relevance of adolescent treatment recommendations does not depend on whether elsewhere in the document another unrelated recommendation is supported by a systematic review.

  Notably, using this definition led to the identification of only five guidance documents. By comparison, two recent systematic reviews of guidelines on the same topic identified 12 and 23 clinical guidance documents, respectively. At the same time, these unusual criteria perfectly match the process found in the World Professional Association for Transgender Health (WPATH) and Endocrine Society guidelines. Those guidelines include some systematic reviews of a narrow set of outcomes in adults, but they do not reference any systematic review of the evidence on pediatric hormonal interventions and instead rely on discussing selected studies.

• Omission of relevant guidelines. As with their review of systematic reviews, the failure to search gray literature, including the websites of public health authorities, was a serious omission in the search strategy. The Review is notably missing the Swedish and Finnish national guidelines—the only two clinical guidance documents recommended for implementation by a recent systematic review of guidelines from the U.K.

  Among the identified guidelines, two are well-known English-language guidelines by WPATH and the Endocrine Society. Three others are lesser known: a position statement on menstrual suppression by the American College of Obstetricians and Gynecologists, a position statement from the European Society for Sexual Medicine, and an outdated 2013 German guideline for children and adolescents with gender dysphoria.

• Inconsistent adherence to eligibility criteria. Inexplicably, among the five "qualifying" guidelines, two do not qualify even under the University of Utah’s modified eligibility criteria. Specifically, neither the European Society for Sexual Medicine Position Statement nor the 2013 German "S1" guideline for children and adolescents meets the requirement of having performed a systematic review for any part of the clinical guidance document. Further, if the University of Utah researchers considered such consensus-based documents eligible, then additional clinical guidance documents—such as the influential American Academy of Pediatrics' 2018 Policy Statement and the Australian Standards of Care—issued by "recognized medical authorities" but lacking any systematic reviews— should also have qualified.

• Failure to engage with the Cass Review. The 2024 Cass Review—a seminal report providing a foundational analysis of pediatric gender medicine that informed the adoption of more cautious treatment policies for gender dysphoria in the U.K. for children and adolescents—appears to match the criteria for a clinical guideline set out by the University of Utah. It was commissioned by a recognized medical authority, NHS England, and was supported by a series of systematic reviews. The Cass Review was published five months before the Evidence Review was submitted to Utah DHHS and more than a year before its eventual publication. It is unclear why this important work was not included in the Utah analysis.

• No guideline quality assessment. Rather than using a validated instrument, such as AGREE II to assess the methodological quality of clinical practice guidelines, the authors relied on the issuing body’s status as a “recognized medical authority,” substituting an appeal to authority for formal appraisal (p. 19). The reputation of the guideline developer is a biased criterion that is unrelated to the quality of the guidelines. There are several examples of guidelines developed by reputable organizations that do not meet expected quality criteria. No established tool for assessing clinical guideline quality treats institutional prestige as a validated proxy for methodological rigor.

  Evaluating guideline quality is crucial for interpreting recommendations, as it helps readers understand how trustworthy they are. The AGREE II tool allows evaluation of the entire guideline process, including scope and purpose, stakeholder involvement, methodological rigor of development, clarity of presentation, and applicability of the recommendations. It also assesses editorial independence, including funding and the disclosure and management of financial and nonfinancial competing interests. It is unclear why this assessment was not performed.

• Failure to independently analyze levels of evidence (LOE) supporting recommendations. A stated objective of the Evidence Review's analysis of guidelines was to assess the "levels of evidence" (LOEs) that support guideline recommendations (p. 3; First Contract Amendment, Attachment D). The University of Utah researchers did not perform an independent assessment of LOE ratings for the guidelines, instead reporting only what the guidance documents themselves reported. Concerningly, the researchers uncritically repeat WPATH’s misrepresentation of the strength of evidence in Standards of Care 8, recording that “strong” treatment recommendations were supported by (“correlated with”) "high" quality evidence (p. 298). An independent assessment of LOE for WPATH would have revealed that the quality of evidence behind its recommendations is “low” or “very low.” Court documents appear to confirm that in at least one Standards of Care 8 chapter cited in the Evidence Review, WPATH graded recommendations as strong without regard to the level of evidence.

Rather than formally appraising guideline quality, the researchers focused on summarizing recommendations from English-language guidelines, with an emphasis on those issued by WPATH and the Endocrine Society, underscoring the following key points:

• Puberty blockers and cross-sex hormones are the standard of care. The researchers present treatment with puberty blockers and cross-sex hormones as the undisputed standard of care for gender-dysphoric youth, emphasizing that “Guidelines providing hormonal therapy recommendations for TGNB [transgender and nonbinary] adolescents are generally in agreement about recommended therapies and treatment approach” (p. 37). In addition to inappropriately framing these interventions as evidence-based, the Utah researchers frame them as “medically necessary” (p. 36).
• Children are not eligible to receive hormonal interventions. The summary emphasizes that prepubertal children are not eligible for hormonal interventions for gender dysphoria. This is an accurate but trivial claim, as there is no biological basis for altering sex hormone production in bodies that do not yet produce such hormones in any meaningful quantity. This claim is often used to ease ethical concerns about hormonal treatment in children just starting puberty, who may be as young as 8 or 9—at which point they are referred to as “adolescents.” Notably, the Utah Review adopts the definition of “children” as youth who have not begun puberty, and “adolescent” for those who have begun puberty, regardless of age (p. 4).
• Both puberty blockers and cross-sex hormones can be provided to adolescents at the first signs of puberty. The Review emphasizes that according to the WPATH guidelines, not only puberty blockers but also cross-sex hormones can be administered at the earliest pubertal signs (potentially “adolescents” as young as eight or nine years of age).

Despite not having appraised the quality of the guidelines, the Evidence Review asserts that pediatric gender transitions should remain available in part because "high-quality guidelines are available to guide qualified providers in treating pediatric patients who meet diagnostic criteria."

Of note, several peer-reviewed guideline quality appraisals that used standard tools found that all guidelines in the field of pediatric gender medicine, with the exception of the Finnish and Swedish national guidelines, are of poor quality, highly interdependent, and circular. The most recently published appraisal of WPATH's guideline quality found that recommendations relating to adolescents have serious limitations in “scientific and methodological rigor, applicability, and transparency in managing competing interests.” The researchers concluded that “uncritical adoption or endorsement of WPATH’s guidelines may result in a disservice or even harm to this vulnerable population.”

Overall, the Utah Review’s guideline analysis is incomplete and methodologically weak. By omitting key guidance documents, failing to appraise guideline quality, and substituting institutional authority for methodological evaluation, it does not provide a reliable basis for evidence-based decision-making.

8. Desistance and Regret

The Utah Legislature requested an analysis of “rates of desistance and time to desistance.” The University of Utah researchers broadened the request and presented their findings in a section titled “Objective 4: Persistence, Desistance, and Regrets” (p. 83-89).

The analysis did not directly address the question of desistance and detransition, but it did conclude that “there is virtually no regret associated with receiving the treatments, even in the very small percentages of patients who ultimately discontinued them” (p. 91). However, a close reading of the section reveals that the analysis is inadequate at every stage. We highlight some of the most serious limitations below.

• Failure to assess detransition/desistance due to omission of relevant evidence. The researchers made no attempt to analyze studies on desistance, commonly defined as cessation of the desire to undergo medical transition prior to medicalization. Nearly a dozen studies report desistance rates between 61% and 98% among individuals with prepubertal onset of gender dysphoria who are not medically treated during adolescence, with desistance typically occurring before mature adulthood and a homosexual orientation being a common adult outcome.

  Furthermore, nearly all the studies related to detransition among recently presenting patient groups had been omitted by the analysis, including survey-based detransition research by Littman (2021); the U.K. detransition and regret analyses using clinic data (Boyd et al., 2022; Hall et al., 2021); and a study of adolescent hormone discontinuation rates from medical and pharmacy records from the US Military Healthcare System reported by Roberts et al. (2022). These studies indicate that the rate of medical detransition among those hormonally treated in adolescence and young adulthood is as high as 10%–30% within several years after initiating transition.

  Two major problems in the study search and inclusion strategy led to this oversight. First, the search strategy only looked for studies of adolescents, whereas desistance/detransition and potentially associated regret are more likely to be found in studies of adults, due to a widely recognized “honeymoon” period that can follow the initiation of hormonal interventions. Second, despite DRRC’s contract with Utah DHHS (First Contract Amendment, Attachment D) and the Evidence Review’s own research objectives (p. 3) including an examination of “rates of discontinuation,” the outcomes of “detransition,” “desistance,” and “regret” were excluded from the “high priority” outcome list that governed which studies would undergo analysis. As such, the studies most relevant to the question at hand were either never identified or were identified but relegated to bibliography-only—with no chance to inform conclusions.

• Unreasonably narrow definition of “regret.” It is well known that most studies reporting detransition and regret suffer from serious methodological limitations, including short follow-up periods, high dropout rates, and narrow definitions of regret. The University of Utah DRRC team does not acknowledge these limitations.

  For example, the researchers’ conclusion of “virtually no regret” is supported by a reference to a large Dutch study whose definition of “regret” likely excludes many individuals who experience regret following treatment (p. 91). Using the study’s definition, well-known detransitioners—such as Chloe Cole and Fox Varian in the U.S., who are suing their providers, or Keira Bell, who was involved in judicial review litigation in the U.K.—would not be classified as having experienced regret. This is because they never underwent the removal of ovaries, whereas under the definition used by the study, individuals would be counted as experiencing regret only if they had undergone gonad removal (ovaries or testes) and subsequently restarted natal-sex hormone supplementation.

• Errors in study summary statistics. As in other tables throughout the Evidence Review, there are numerous errors in these data tables. For example, when describing the Wiepjes et al. (2018) study cited above, the researchers report the following incorrect statistics on pubertal suppression and cross-sex hormone therapy: “No cases of regret were observed among the 1,360 individuals who were first seen before the age of 18 years. 1.9% of adolescents who started PS [pubertal suppression] (n=812) stopped PS and did not start HT [hormone therapy]” (p. 89). However, the number of adolescents that commenced pubertal suppression was 333, not 812. Further, regret rates pertained only to those adolescents who were on cross-sex hormones and underwent gonadectomy, a group numbering 309, not 1,360.
• Lack of assessment of quality/certainty of evidence. As with all other analyses in the Evidence Review, no formal evidence synthesis was offered that summarized the body of evidence at the outcome level and assessed the findings for certainty. Instead, readers are directed to Table I.26 (pp. 83–89), which lists the (frequently inaccurate) characteristics of 32 studies. This table does not, however, specify which studies informed the conclusion of “virtually no regret,” nor does it caution readers about the narrow definition of regret or the degree to which short follow-up duration and high drop-out rates can bias estimates downward or otherwise distort conclusions.

The inadequate analysis of desistance, detransition, and regret is difficult to defend in light of the critical importance of these topics both to the commissioning body and to the patient population, which has undergone a marked epidemiological shift in recent years—and in light of rapidly growing reports of detransition, including detransitioners who regret the gender transition interventions they received in adolescence. It is even more problematic that the Utah DHHS Recommendation Report, which accompanied the Evidence Review, failed to recognize the profound problems in the analysis and instead bolstered the analysis’s credibility by suggesting that the DRRC conducted a “systematic review” providing “a complete summary” of persistence, desistance, and regret (Recommendation Report, p. 7).

9. Interruption of Normally Timed Puberty

The Utah Legislature explicitly requested that the evidence review assess the “short-term and long-term benefits and harms of interrupting the natural puberty and development processes of the child.” The Utah Review left this request unaddressed and provided no explanation.

Below, we highlight several key problems associated with the decision not to conduct this important analysis.

• Lack of consideration of evidence from basic sciences and physiology. Puberty is a time-sensitive, hormone-driven developmental process affecting bone, brain, fertility, and metabolism. Disrupting it may produce cumulative or delayed effects that short-term observational studies miss. For this reason, evidence of the potential harms of endocrine disruption to physical health and overall development must be supplemented by knowledge from basic science and physiology relating to endocrine mechanisms and developmental timing. It is concerning that the Utah Review did not consider this analytic component.
• Incomprehensible exclusion of “fertility” as an outcome of interest. As explained elsewhere in our analysis, the University of Utah researchers’ explicit decision not to consider “fertility” a “high-priority” outcome on the grounds that hormone-related fertility harms are already a “known risk” (Recommendation Report, p. 4) is methodologically incomprehensible. This decision ensured that one of the best understood and most ethically problematic harms of pediatric medical transition—likely infertility—was unaccounted for in the Evidence Review’s analysis of benefits and harms.
• Failure to consider sexual function as an outcome of interest. Early puberty suppression followed by cross-sex hormones may impair sexual function. The lack of attention to this important aspect of human health is concerning.
• Lack of consideration of the neurodevelopmental effects of interrupting normal puberty. Puberty drives structural and hormonal brain maturation, including synaptic pruning and myelination. The University of Utah researchers made no attempt to assess what is known about the effects of suppressing this biological developmental process on the brain.

The Evidence Review offers no examination of these or numerous other risks and potential harms of interrupting normal puberty. Nor does it explain why the researchers chose not to undertake this required analysis, which is entirely feasible and was conducted in the HHS Evidence Review. Instead, the University of Utah researchers appear to sidestep this analysis entirely based on the assertion that “conditions like GD [gender dysphoria] … have no other effective interventions” (p. 63). This highly consequential claim is offered without citation or analysis and is unjustifiably presented as an established fact.

10. Interpretive Bias in the Evidence Review