In a media interview, Bränström, the primary author, suggested that the study results lent “strong support” for providing “gender-affirming” care, while Pachankis, the study-co-author, issued a statement saying, “no longer can we say that we lack high-quality evidence of the benefits of providing gender-affirming surgeries to transgender individuals who seek them.” The study and its conclusions were disseminated by Medscape, relied on by millions of clinicians and researchers worldwide, and has become the basis for Medscape's Continuing Medical Education class (CME). The US-based Association of Schools and Programs of Public Health, which describes itself as the “voice of academic public health,” also disseminated the study and its findings to its member public health programs. To date, the study has been referenced by at least 9 other studies as evidence of benefit of “gender-affirming” surgeries (Table 1).

TABLE 1. Publications Found Citing Bränström & Pachankis (2019).

The methodology used by the authors was deemed inadequate to reach the conclusions supporting surgical treatment for gender dysphoria (4-10). The multiple methodological issues identified by researchers from Sweden, Norway, US, and UK, included:

• Single year short-term outcomes. While the study was interpreted by many as a 10-year follow-up, it is effectively a single-year, short-term outcome study. Of the individuals who underwent surgery (n=1,018), fewer than 2% had surgery 10 or more years ago (n=19), while over 75% had surgeries within the past three years (n=772). Analyzing single-year outcomes from different initial years is fraught with methodological problems and is not equivalent to a longitudinal study.

• Failure to account for the recent rise in co-occuring mental health problems. In recent years, there has been a general increase in mental health morbidity in the Swedish population, as well as a noted shift in patients presenting for gender dysphoria treatments, with most patients now adolescent females with significant mental health comorbidities (11,12). Thus, the better mental health of the older cohorts could explain why those who transitioned further in the past utilized fewer mental health services in 2015, as compared to the more recent transitioners.

• No comparison to pre-surgery mental health status. Rather than comparing the patients' mental health services utilization in 2015 to utilization pre-surgery in order to estimate the impact of surgery, the authors instead compared it to the utilization of mental health services during the year of the (last) surgery. Choosing the year of the surgery as the baseline is highly problematic, because surgery in itself may have been associated with a temporary increase in mental health services utilization (such as additional routine evaluations or a result of surgery-related stress). Thus, it is impossible to determine whether the detected reduction in mental health utilization in 2015 compared to the year of surgery is a true reduction in one’s baseline mental health utilization, or if it is merely reflecting a return to the baseline, following a potential short-term spike in mental health needs brought about by the surgery itself.

• Unaccounted for high loss to follow-up. It’s unclear why the researchers only identified a total of 19 patients who underwent surgeries before 2005, since another study using the same national registry found 324 patients in the years prior to 2004 (13). It’s highly problematic if over 90% of postoperative gender dysphoria subjects were lost to follow-up. It's surprising that the reasons for such a large discrepancy were not addressed by the study authors, especially since both studies were conducted by the Karolinska Institute and they studied the same population. The reasons for this large discrepancy must be clarified. Prior studies demonstrated that post-surgery transgender patients suffer from elevated mortality due to cardiovascular disease and suicide (13). To the extent that even some of this loss of postoperative transgender patients is due to premature death, this large loss of study subjects may have masked adverse outcomes.

• Questionable proxy for mental health. The authors used outpatient mental health visits, prescriptions for mental health conditions, and hospitalizations following suicide attempts as surrogate measures of mental health. However, the use of these measures is problematic. Outpatient visits recorded as related to mental health may or may not signal the presence of mental health issues. For example, mental health assessments were required prior to surgery; conversely, visits not recorded as related to mental health could signal mental health problems, especially those related to sleeping disorders or substance use. The inclusion or exclusion of different drug prescriptions was arbitrary and inconsistent; histamine blockers were judged related to mental health issues, whereas antipsychotic medications were not. Also, only suicide-related mental health admissions were considered, ignoring all other mental health-related admissions. The suicidality measure is particularly questionable, since only suicide attempts requiring hospitalization were analyzed, with no accounting for patients who actually died by suicide or those whose suicide attempt did not result in a hospitalization.  

• Failure to correct for multiple hypotheses testing. The observed reduction in mental health treatments utilization associated with surgeries was small - an average 8% reduction per year - and was statistically significant by only a thin margin (adjusted odds ratio=0.92; 95% CI= 0.87-0.98). Further, this was the only statistically significant finding among several that the authors report to have tested. However, the authors did not appear to have performed a standard statistical adjustment for multiple hypotheses testing. This is a critical concern, since this necessary correction would likely have rendered this already weak result not statistically significant. The increasing frequency of “fishing expeditions,” made possible by large datasets and sophisticated software, has led to a well-recognized crisis of replication in healthcare research, with many findings of statistical significance that cannot be reproduced by other researchers. Reducing false positive findings is crucial when these findings are used to guide decisions about medical interventions that carry substantial health risks and that are irreversible.

• Failure to include a control group. The authors failed to include a control group - comparing gender dysphoric patients who underwent surgery with those who did not. The Swedish database contains comprehensive information for both cohorts, and such a comparison should be performed before any conclusions about the benefits of gender affirming surgeries are made.

After these and a number of other limitations had been pointed out by several researchers, the AJP editor commissioned an independent statistical evaluation. The independent expert concurred with the concerns raised (3), and recommended that the answer to the key study question - whether surgeries improve mental health outcomes - should be clarified by creating a control group and reanalyzing the data. Complying with the independent statistical expert’s recommendations, the study authors created two equally-sized groups of patients diagnosed with gender dysphoria, the "surgery" and the "no-surgery" groups, which were matched on key demographic characteristics. When the two groups were compared, no significant differences were found in any of the mental health utilization measures used as a proxy for overall mental health (14). These findings are illustrated in Figure 1 above.

As a result, the AJP issued a correction, which states, “the results [of the reanalysis] demonstrated no advantage of surgery in relation to subsequent mood or anxiety disorder-related healthcare visits or prescriptions or hospitalizations following suicide attempts,” and “given that the study used neither a prospective cohort design nor a randomized controlled trial design, the conclusion that “the longitudinal association between gender-affirming surgery and lower use of mental health treatment lends support to the decision to provide gender-affirming surgeries to transgender individuals who seek them” is too strong (2).

The authors summarized the key findings in Table 1 of their new paper, “Toward Rigorous Methodologies for Strengthening Causal Inference in the Association Between Gender-Affirming Care and Transgender Individuals’ Mental Health: Response to Letters” (14).

1. "Gender-affirming" surgeries may improve mental health
   It is possible that despite the fact that the “surgery” group, on average, did not show a reduction in utilization of mental health services compared to the “no-surgery” group, some patients may have benefited from surgery. The study authors correctly point out that the comparator group may include gender dysphoric patients who did not seek surgery and therefore it cannot be concluded that the lack of difference between the two groups signals the lack of benefit of surgeries for those who seek them.
2. "Gender-affirming" surgeries may worsen mental health
   It may be argued that not only has the re-analysis invalidated the original conclusion of proven benefits of “gender-affirming” surgery, but it has identified a possible worsening of mental health following surgery. The control group represents a strong comparator: patients in both the control and the intervention groups had confirmed DSM diagnoses of gender dysphoria, which takes at least a year to obtain; further, the two groups were matched using age, legal gender, education, and country of birth. Worse mental health outcomes during the peri-operative period (the year of surgery) noted in the original study, and the nearly doubling of serious suicide attempts in the “surgery” cohort compared with the “no surgery” cohort, which emerged during the re-analysis of the data, are consistent with this interpretation.
3. The current study is insufficient to draw conclusions about the impact of "gender-affirming" interventions on mental health
   Given the challenges of conducting quality research in the absence of randomized control groups, and based on the study’s chosen methodology, it can be argued that the data presented in the study can provide no reliable information whether “gender-affirming” surgeries improve, worsen or have no effect on mental health.

SEGM is concerned by the renewed claim that despite the reanalysis, which showed no difference between the control and intervention group, the original study’s conclusion of continued improvements in mental health after surgery stands (14). SEGM does not believe that the data presented by the authors support this claim, because as yet unaddressed methodological limitations continue to confound the original study’s findings. 

It is SEGM’s view that the longitudinal association between time from surgery and improved mental health outcomes, cited in the original analysis, is confounded by serious methodological limitations, and cannot be considered valid until these limitations are addressed. Notably, contrary to the authors’ claim, the analysis failed to control for differences in the baseline mental health status of the cohorts of patients seeking surgery. The well-documented increase in mental health burden of the more recent patient cohorts (11,12) alone can explain why those who had surgeries a decade ago have better mental health status in 2015, than those with more recent surgeries. It is critical that the analysis corrects for the underlying mental health of each patient cohort before any claims of “a reduction in mental health treatment as a function of time” are posited, let alone accepted. Other issues raised in the letters also must be addressed, especially in the context of the fact that a reanalysis using a control group failed to lend support to the original finding.

SEGM is also concerned that the surgery group experienced nearly twice as many suicides compared to the non-surgery group, based on 2015 numbers (13 and 7 suicide attempts, respectively). While not statistically significant, this finding deserves additional analysis. Since suicides are somewhat rare events, the reanalysis should include a longer-term, rather than a single-year, follow-up, in order to properly power the study. 

The number of critical letters to the editor and the need to correct a core conclusion of a published paper highlights the importance of stringent methodology and rigor in medicine. Many of the original study’s shortcomings should have been obvious and might have been avoided by better protocol design. Nevertheless, this episode shows the self-correcting mechanism of "the many eyes of science." It underscores the need for researchers to engage in scientific debate in order to further the quality of scientific inquiry in the rapidly changing field of gender medicine. 

The need for quality information in gender medicine is particularly crucial, given that the majority of patients seeking hormonal and surgical interventions for gender dysphoria are now young people, most of whom are females with adolescent-onset of gender dysphoria and co-occurring mental health difficulties (15). Today, irreversible hormonal and surgical interventions are routinely offered as first line treatment to this novel and poorly-understood cohort of patients. 

It is significant that the largest population study conducted to date, which assessed the mental health benefits of hormonal and surgical treatments to alleviate gender dysphoria, did not provide evidence that these interventions were accompanied by a reduction in subsequent mental health treatments or serious suicide attempts. Substantial research is needed in this vitally important area of medicine to establish the type and timing of interventions for the increasingly varied presentation of gender dysphoria worldwide.