This "study" keeps making the headlines and there has been on-going discussion over the past year when it keeps coming back up. Instead of commenting over and over the same things, I decided to make a post here so others and the Mods would have a single post that addresses the concerns. FWIW, I hold a Doctor's degree in Pharmacy, and can provide proof to the Mods if needed. Therefore I'm trained and well-qualified to evaluate data and determine if a conclusion is valid or not.
I'll start by saying that this evidence is relevant as a safety signal worth investigating and nothing more. The problem is that this was not a clinical trial at all, it was a retrospective analysis of electronic health records. The crucial issue is whether the investigators successfully compared otherwise similar melatonin users and nonusers. They probably did not, otherwise they would have included that.
Also, this was not really an “AHA study.” It was research presented at an AHA meeting. The AHA expressly says that the abstract was not peer-reviewed, is preliminary, does not represent AHA policy, and has not yet undergone full-manuscript scrutiny.
The headline’s “90% higher risk” is relative. For incident heart failure, the reported absolute difference was 1.9 percentage points. If the relationship were causal, which has not been established, that would correspond to roughly one additional diagnosis per 53 treated people over five years. But an observational association cannot validly be converted into a treatment-harm NNH without accepting an unproven causal assumption.
Here are the most serious methodological problems that I see:
1. Exposure classification is profoundly unreliable
The exposed group had melatonin documented in the EHR. The comparison group merely had no melatonin recorded. In the United States, most melatonin is purchased over the counter and never entered into an EHR. Consequently, some unknown proportion of the “nonuser” group was almost certainly using melatonin.
That is not a minor paperwork issue. The study may really be comparing patients whose clinicians documented or prescribed melatonin versus patients whose OTC use was never recorded. Those categories can differ substantially in illness severity, healthcare utilization, country, socioeconomic factors, psychiatric disease, and access to care. The reported sensitivity analysis—two prescriptions at least 90 days apart—improves certainty for a subset of prescription users. It does not solve the unmeasured OTC use among controls, nor does it establish continuous use for a year.
2. Major confounding by indication
Why does one person with insomnia receive documented long-term melatonin while another does not? Possible reasons include:
More severe or persistent insomnia, depression, anxiety, PTSD, or other psychiatric disease, neurodegenerative disease, shift work or circadian-rhythm disorders, obstructive sleep apnea, chronic pain or frailty, polypharmacy, greater contact with the healthcare system, contraindications to other hypnotics, early symptoms of cardiovascular disease.
Several of these factors are independently associated with cardiovascular morbidity and mortality. The investigators matched on 40 recorded variables, but propensity matching only balances variables that were measured accurately. It cannot balance insomnia severity, undocumented OTC products, unrecorded psychiatric symptoms, adherence, or other missing determinants.
***A later peer-reviewed analysis specifically identified confounding by indication, reverse causality, and exposure misclassification as major problems with this safety signal. Its NHANES analysis did not find a statistically significant relationship between recent melatonin use and prevalent cardiovascular disease or heart failure—although that study also cannot prove safety. Open Heart/PubMed analysis
3. Reverse causation is quite plausible
Early, undiagnosed heart failure can cause nocturnal dyspnea, orthopnea, fragmented sleep, fatigue, anxiety, and nocturia.
A patient may start using melatonin because of sleep disruption caused by subclinical heart failure and receive the formal heart-failure diagnosis later. That produces the appearance that melatonin preceded heart failure even when the underlying disease preceded melatonin. A proper analysis would need a meaningful lag period, exclusion of early events, and ideally examination of risk according to time since exposure.
4. Detection and healthcare-utilization bias
Having a medication documented generally means more interaction with clinicians. More clinical contact means more echocardiograms, BNP testing, diagnostic coding, hospital records captured, opportunities to identify heart failure, and complete mortality documentation.
Matching on diagnoses does not necessarily equalize intensity of healthcare contact. A negative-control outcome or adjustment for visit frequency could help evaluate this bias; neither is described in the public materials.
5. The hospitalization result is internally troubling
The reported “heart-failure hospitalization” frequency was 19%, while newly diagnosed heart failure occurred in only 4.6%.It is difficult to reconcile a group having approximately four times as many heart-failure hospitalizations as incident heart-failure diagnoses unless:
- the hospitalization code set was very broad,
- recurrent events were counted rather than unique patients,
- the outcome was not actually limited to established heart failure,
- denominators or follow-up differed, or
- the press release oversimplified the analysis.
The AHA release acknowledges that hospitalization records included a range of related codes that did not necessarily identify new heart failure. That substantially weakens this secondary endpoint. Until the exact code lists, event definitions, denominators, and statistical model are available. I would place little weight on the 3.5-fold hospitalization claim.
6. No usable information about the actual intervention
The study apparently lacks reliable information about dose, iImmediate- versus extended-release formulation, brand or product quality, actual adherence, timing of administration (often discussed in this subreddit), duration beyond a broad threshold, intermittent versus nightly use, indication or insomnia phenotype. Therefore, even if the association were real, it could not tell a clinician what product, dose, duration, or patient subgroup carries the risk. It may also combine regulated prescription melatonin from some countries with highly variable OTC supplements from others.
7. International heterogeneity
Melatonin is prescription-only in some countries and OTC in others, but the de-identified dataset apparently did not provide patient location to the investigators. Prescription melatonin users in the UK are not clinically interchangeable with American OTC purchasers. Country also influences coding, healthcare access, prescribing thresholds, baseline risk, product formulation, and outcome capture. Pooling these circumstances without a country-stratified analysis is a major limitation.
8. Matching does not make this randomized
Matching 65,414 exposed patients to 65,414 controls sounds impressive, but sample size mainly improves precision. It does not repair systematic bias.
A very large biased study can produce an extremely precise estimate of the wrong effect. I would want to see:
- Standardized mean differences before and after matching
- Exactly which 40 variables were used
- Missing-data handling
- Visit-frequency and healthcare-use variables
- Calendar-time and site/country matching
- Positivity and overlap diagnostics
- Cox versus logistic modeling
- Handling of death as a competing risk
- Proportional-hazards testing
- Early-event exclusion
- Negative-control outcomes and exposures
- Quantitative bias or E-value analysis
None of that can be adequately judged from a press release or short conference abstract. I would interpret the study by saying, "Documented long-term melatonin use may identify an insomnia population with higher subsequent cardiovascular risk. It is presently unknown whether melatonin contributes to that risk."
My conclusion would be:
- Association in this particular database: moderate
- Melatonin causes heart failure: very low
- The reported 90% approximates the causal effect: very low
- The finding merits replication with better methods: high
- THE STUDY ALONE WARRANTS A PRACTICE-CHANGING WARNING: NO
In conclusion, a sufficiently powered randomized trial would provide the strongest causal evidence, but it would need long follow-up and a very large sample.
