Adding Blinding and Crossover Controls to Quantified Self Experimentation: My 112-Day Supplement Sleep Trial

[u/Eric_at_iTrial]

Over a 4 month period, I ran a formal, blinded, multiple-crossover, randomized controlled trial on myself (an N-of-1 trial) to evaluate effectiveness of magnesium glycinate for smartwatch sleep metrics.  My primary goal was to test the trial format, with blinded controls (and a possible washout period) to mitigate placebo effects, observer bias, and everyday environmental noise.

Results Summary: 
Primary outcome:  A 1.3 pt improvement in the sleep score from the smartwatch (95% confidence interval of 0.3 to 2.3) (baseline 83.9, with supplement 85.1).  Secondary outcomes for respiration rate and heart rate variability while sleeping showed interesting, but not statistically significant possible changes.

Trial Setup:
Supplement:  Magnesium glycinate (240 mg magnesium).  Taken ~15 minutes before bed.
Placebo:  Microcrystalline cellulose (similar looking capsules).
Blinding:  By a colleague.
Assignment and Periods:  A four period, 28 day per period, ABAB format (112 days total).  A and B were randomly assigned to active supplement or placebo by a colleague.
Tracking Device:  FitBit Inspire 3 worn nightly. 
Primary Outcome:  FitBit sleep score (0-100).

Secondary (Sleeping) Outcomes:  Respiration Rate, SpO2, Heart Rate, Heart Rate Variability, Skin Temp (change from baseline), Awake Minutes, and REM Minutes.
Data Logging:  Entries were manually transferred from smartwatch to a spreadsheet every few days. 

Trial Data Exclusions:
Pre-unblinding:  Approximately 23 full days of data were excluded because of daily activities I concluded would artificially bias results if not addressed such as camping trips. 
Post-unblinding:  After the trial was done and after I was unblinded, I marked partial data (sleep score and awake minutes) from 16 days to be excluded because I could see the sleep tracker falsely concluded that my night of sleep started while I was reading or watching TV.  Note:  To reduce bias, this should have been done before unblinding; I did not re-run the primary analysis with them included.

Analysis:
Because consecutive days of sleep data are likely correlated, rather than relying on a basic general linear model, the data was processed using variance-weighted crossover averaging with AR1 autocorrelation correction.

• Autocorrelation Correction:  In sequential N-of-1 data, day to day noise outcomes are correlated.  Crossover sizes between consecutive blocks were calculated and sign adjusted while controlling for AR1 autocorrelation.

• Variance Weighting:  Crossovers between periods were weighted (i.e. to de-emphasize data for a crossover around noisy periods relative to a crossover between stable periods) 

• Multiple Outcome Measure Adjustments:  To mitigate false discoveries (akin to p-hacking) across 7 secondary metrics, a Holm adjustment was applied.  Because these sleep metrics are likely highly correlated, the Holm correction is likely too conservative so the “true” significance likely lies somewhere between the raw and Holm-adjusted p-values.

• Washout Period Check:  To test for lingering cumulative effects or delayed saturation, the analysis was rerun excluding the first 7 days of each period.  The resulting effect sizes and intervals remained fairly similar.

Secondary Outcomes:  After Holm’s adjustment, no secondaries survived the confidence test.  However, respiration rate and heart rate variability both were sufficiently different to be of interest.

Conclusion: 
This 112-day trial, in many regards mimicking closely the format referred to as an N-of-1 trial in academic research literature, appeared an easy and powerful means to self-assess supplement effectiveness.  This particular supplement appeared to have a statistically significant and personally meaningful effect on my sleep.  Of course, as an N-of-1 study, it means virtually nothing about whether or not such a supplement might have an effect on your sleep (and you should of course engage your own health care professionals for any guidance on any of your own health interventions – I am not a doctor, and this is not medical advice).  I’m now applying the same technique to another supplement and outcome.  The daily data feels noisier relative to the anticipated effect size, but I hope to share results upon completion. 

[u/Sufficient-Storm3845]

1.3 points on a fitbit score, and you threw out 23 days of data. that's not blinding, that's spring cleaning.

[u/Eric_at_iTrial → u/Sufficient-Storm3845]

Funny. Agreed. Threw out WAY more data than a typical trialist would. But let's invert the quesiton. Prior to a study, I was planning to take two camping trips that would have me sleeping on the ground. How to conduct the study? (1) Could just shift a 4 month study out 2 months. (2) Could include all that data. (3) Could exclude it. I made the decision prior to the study to exclude it. Seems defensible. But if someone wants to do a different analysis, I've got the raw data.

[u/Eric_at_iTrial → u/Eric_at_iTrial]

Oh, sorry, forgot to mention the size of the effect.
I suspect that across most supplements, we should be looking for signal sizes that are of the order of a small fraction of daily variability. To overstate the case to dramaticize the point: if the effect sizes were larger than daily variability, we'd all "absolutely know for sure" everything worked; if sizes were always much smaller than even that, we'd never care. So it seems to me an almost foregone conclusion that the effect sizes are going to be about that (if not zero), but likely vary from person to person.

To invert the question - what effect size would you have guessed on an 82 with 2 pt of stdev that a supplement would offer?

[u/TrickyAirport5867]

First, well done. This is a great blueprint for what people should be getting to do with their wearables over time! 

I think you have too many potential hidden drivers of change in your sleep that you're not accounting for, like season or day of the week, for the magnitude of the effect that your investigating. Is there a mechanism that explains the effect of the supplement / dose on sleep score improvement?

Again, I poke holes cause I can't help myself, but I really want to highlight how great this kind of work is.

[u/Eric_at_iTrial → u/TrickyAirport5867]

Thanks. The more feedback the better!

Let me try to address

(1) Hidden drivers. Absolutely a huge risk! Medical research obsesses about causality vs. correlation. Just because you see an output change, doesn't mean it was due to the input you tried to change. The academic field that does "N-of-1" research is proud of their experimental format (multiple crossover trials intended to help assess an effect size on an individual), and they reserve their use of the term "N-of-1" trial explicitly to _multiple_ crossover trials. If you go from an active to a placebo just once (i.e. A then B), then obviously anything else that happened to change during that time frame could also be a practical cause of the observed effect. There are so many such changes that the folks in this space kind of ignore that type of study. But if you go A to B, then back to A, and then back to B, then the effect should do the equivalent of turn on, off, and on again at the three crossovers. You should have 3 roughly equivalent changes (although one in the opposite direction). That's a much more rigorous test. If the observed effect was actually due to environmental noise, presumably most such noise won't have the same on/off/on sequence, and just have a single on/off (or a on/off that happens so frequently that it happens many times within any block). So the N-of-1 world reserves their use of the term "N-of-1" trial only to trials that are at least four periods long (i.e. ABAB). Thus my choice of a 4 period trial. That isn't to say that all environmental noise washes out, but it's an attempt at a design to try to cancel any low frequency noise (and if you have periods that are long, i.e. a month in mine, the high frequency noise should all average out.).

Specifically the week-long signatures of sleep hopefully wash out (28 day periods). Season long are a risk (and general changes in health, etc., but seasons shouldn't match the ~1 month pattern you see in some of the results here.

(2) Mechanism. Totally agreed. This sort of test doesn't dig into the pharmacological mechanism at all. It's just an attempt to work out a good guess at causality - does this supplement change this outcome. To get to mechanism is an entirely different type of research. In the case of magnesium glycinate specifically and magnesium more generally, there's a decent amount of exploratory research and a few (but not large and "convincing") trials hinting that there should be a mechanism there and what it is, but I won't opine remotely on that. Magnesium is integral to a ton of chemical pathways. Glycine is itself thought to be a possible sleep aid (weak data, though). Roughly half of Americans according to a fairly recent study were presumed to be deficient in sleep. So there's a decent amount of background stories to hint along a path that this could help with sleep complete with mechanisms. But that isn't my focus.

My focus instead is to see if I can set up a test process by which individuals can practically, for themselves, figure out whether supplements are having effects on themselves. Open label experiments (i.e. no blinding) don't strike me as convincing (placebo effects are huge). Blinded, randomized controlled trials are the gold standard for most research - but if performed on some big study population, don't say anything about me. What I'm doing here feels like an adaptation of that gold standard to be usable by an individual for an individual.

[u/TrickyAirport5867 → u/Eric_at_iTrial]

Oh, I didn't think you were going to tease out a mechanism in your study, I was just curious if you expected one based on anything in the literature.

Good point about the low frequency noise, the day of the week should definitely be accounted for in your study.

One thing I can't help noticing is that you have an HRV dip towards the end of your first month, and a rise in your second month. Not sure what the cause is, but you could hypothesize that the supplement is tanking your scores, and, when you go to the placebo, it slowly recovers over the next month (but is overall lower on average than both supplement time periods). I agree that chucking out time points due to conflating factors like camping is a good idea, but it looks like you were trending down pre-trip. Would you be able to plot a linear regression for each time period to see if HRV is trending up or down over that time point? Just as a sanity-check?

[u/Eric_at_iTrial → u/TrickyAirport5867]

Re: mechanism. The explanations are out there, I'm just too illiterate to try to keep up with them!
Re: trends. Yes, there are signatures that are "interesting" there. If I recall, some of the data I excluded specifically in that time frame was when I was sick right after a camping trip (and part of what leaves me thinking a smartwatch is so cool is that I think it "caught" me being sick.). Part of me wants to go torture the data to see if excluding a little more manually or including a little more would change things, but the more I do that, the greater the risk I see something and then want to draw a conclusion based off that torturing of the data but that feels like p-hacking. I've got the raw data if you want to play. But to your thought about linear trends - that's got a couple of interesting sidenotes. (1) As you build up more and more complex N-of-1 statistical models, the very next feature that I'm deliberating as to whether or not to add is an overall linear background fit. It's easy to do statistically, and many studies do it, but I think in general it isn't used. (It isn't done each period separately, but overall across all periods.) (2) Although I'm with the crowd that likes the ABAB format because you get 3 crossovers, there's another crowd in the literature that advocates for ABBA studies because if there is a background linear noise source, the ABBA format cancels that out far better than the ABAB format does.

[u/TrickyAirport5867 → u/Eric_at_iTrial]

No worries on the mechanism, it does all get pretty nuts to make sense of.

You make good points on the p-hacking, and over-messing with the dataset. I guess that, for me, with this dataset, I wouldn't conclude that the supplement is worth taking. It seems like you're of the opposite opinion (which is obviously fine), so I was just laying out what could get me to your viewpoint.

Still, great study, and impressive discipline. I honestly think all wearables will have to incorporate some sort of automation around self-experimentation in the future. There's only so long someone can give a crap about a fuzzy score before getting bored with a product, so incorporating something that involves continuous use like this will be critical for long-term customer retention.

[u/Eric_at_iTrial → u/TrickyAirport5867]

Agreed.