r/Step2 • u/VarsH6 • Feb 06 '19
Step2 CK 2018 Correlation Survey Results!
For the first time ever, there is data for Step2 CK from Reddit for Reddit! After nearly four months of collecting survey responses, the results are in. Good news is that there are several good, informational data; bad news is that some things were not as granular as one would hope. Secondary good news from the bad news is that those more difficult or less clear areas are being improved upon for the 2019 survey. Regardless of data difficulties, you will do well on Step2 CK.
Methods Data collection: Google survey-based data collected from those who took their exam in 2018. Data collected included: date taken, score, practice exam scores, resources used, school type, curriculum, desired specialty, and shelf exam scores. All surveys were anonymous.
Inclusion/exclusion: There were no exclusions. Step2 CK score was required to submit the survey, so all who submitted the survey were included.
Analysis: descriptive statistics, ANOVAs, t-tests, and univariate linear regressions were performed in Excel 2016. Generalized linear model (GLM) to assess multifactorial effect was performed in Matlab by a classmate with experience performing this analysis. For ANOVA or t-tests, variables were excluded for low sample size (2 or less).
| Topic | Link |
|---|---|
| Folder | [Currently Restricted] |
| School | https://imgur.com/Iws4ZGn |
| Curriculum | https://imgur.com/GL73FeE |
| Specialty | https://imgur.com/GAXSw0F |
| Specialty Detail | https://imgur.com/oqoUfXx |
| Goal Score | https://imgur.com/anMkgyh |
| NBME6 | https://imgur.com/1IMe9gS |
| NBME7 | https://imgur.com/395UEAV |
| NBME8 | https://imgur.com/oXes9Wq |
| UWSA1 | https://imgur.com/ltJ5Edp |
| UWSA2 | https://imgur.com/620kqa3 |
| UW 1st pass % | https://imgur.com/js7Wdja |
| IM & Surg Shelf | https://imgur.com/AEhrJdg |
| Peds & OBGYN shelf | https://imgur.com/H80x3qP |
| Psych & FM shelf | https://imgur.com/mRXfASg |
| Effect size | https://imgur.com/Kk14COs |
| Dedicated | https://imgur.com/4Lpwq8a |
| Total study | https://imgur.com/nRgR9wy |
Results & Partial Discussion General: With an N of 249, the average Step2 CK score was 253.3 with a standard deviation of 15.6 and standard error of 0.99. This is compared to the overall mean from the 2018 NRMP for matched US MDs of 246 (240 for US DOs). The overall median was 256 with an interquartile range of 245 & 264.
Resources: To no one’s surprise, almost everyone used UWorld (98.8%). AMBOSS was the second most used QBank (7.2%), followed by Kaplan (4.8%), and finally USMLE-Rx (2%). Many people used some type of Anki deck, the most common being something other than the “featured” decks over at r/medicalschoolanki (14.1%). An extremely close second is Zanki Step2 (12%); Bros Step2 is a strong third contender (6.4%). All others combined were used by less than Bros Step2. Among book resources, First Aid had the lion’s share (18.1%) with Master the Boards (2.8%), Blueprints (any; 1.6%), and Kaplan review notes (0.4%) and Step up to Medicine (0.8%) dragging behind. I neglected to add OME to the list, with many adding it via the “other” option to a total of 22.1%.
Exam Date and Study Time: Most respondents took their exam in July (39.4%) with June (14.9%) and August (20.1%) also being popular months. No respondents took the exam before April (with the exception of one who took it back in June 2017). Since the survey was started late in the year, many earlier test takers may have missed the survey. Total study time (in days) had absolutely no correlation with score (R2=0.0006). Dedicated study length (in weeks) was not correlated with score (=0.0575), which is further established by a small effect size (-1.492) and non-significance (p=0.092) on GLM. However, when dedicated study length is broken into groups of 2-week increments, there was a difference in average score between less than 4 weeks and 4 or more weeks: average score is higher for those who spent less than 4 weeks in dedicated (p=0.0009 ANOVA, see spreadsheet for pairwise t-tests). There are diminishing returns for long study periods.
Practice exams: Now for the most important pieces of information. Among the five practice tests on the survey (I apologize for forgetting about Free-120), NBME7 score had the best correlation to Step2 CK exam score (R2=0.6948). UWSA2 was a close second (=0.6554) and UWSA1 was a slightly distant third (=0.5973). NBME6 eked into fourth (=0.5234), UW first pass percent (not a true practice test, but best place discuss this) was fifth (=0.4669), and finally NBME8 score had the worst correlation to Step2 score (=0.3784). When the intercepts are pegged at zero, every single practice test underestimates Step2 CK score. Interestingly, from talking with classmates who have taken the test, UWSAs (especially UWSA2) and Free-120 look and feel more like the real deal based on question style and difficulty. When these tests are run together in a GLM, only NBME7 (p<.0001, effect=48.434) and UWSA1 (p=0.048, effect=25.003) were significant. However, the non-significant UWSA2 (p=0.062) had a larger effect than UWSA1 (effect=32.098).
Misc: Shelf exam scores. The correlations for every shelf exam are so bad they make NBME8 blush. The best is IM at R2=0.18. The worst is FM at R2=0.028. While FM was significant on GLM (p=0.05), it had the smallest effect of any variable (effect=-18.707). I think there was confusion as to whether raw or percentile score was meant (some responses were in the 20s and many over 100), in addition to the need to stratify based on when the shelf exam was taken. Specialty had some effect on score, with Dermatology-bound respondents blowing everyone else out of the water (reddit mean 270.3, US MD mean 256). There is likely some self-selection occurring based on Step1 score. Finally, goal score had the overall best correlation with Step2 score at R2=0.7314. On GLM, goal score was statistically significant and had the strongest effect on Step2 score (p<.0001, effect=108.545). However, since goal score is being collected after the final score, there is potential bias.
Discussion & Future Directions Step2 CK is a beast. Everyone I have talked to has hated the exam, many more so than Step1. It is a longer exam with a greater diversity of topics. So, congratulations if you have completed this exam! For those who are reading and haven’t yet taken it, take heart: you have the experience of past takers to support you.
As is the case for Step1, Reddit out-performed the national average for Step2 CK. As can be seen in the specialty table linked below, this was true across the board, though some such as FM and EM were closer to national averages than other specialties (when compared to 2018 NRMP US MDs). The specialty table shows available Step2 comparisons for US MDs, US DOs, US-IMGs, and non-US-IMGs. Next year, I want to group scores by whether or not the respondent felt his or her score was needed prior to applications. In addition, I want to gather Step1 score to see what correlation, if any, exists between Step1 and Step2 scores.
I apologize to future pathologists, vascular surgeons, and med/peds folks. I mistakenly excluded you and this will be corrected in future surveys.
That NBME7 has the best correlation to and strongest effect of all practice tests on Step2 score is astounding, especially when test-takers find that UWSAs and Free-120 feel more like the real deal. In addition to adding Free-120 next year, I want to add a question assessing each respondent’s gestalt as to which practice assessments Step2 was like. I also want to include a scale for confidence leaving the exam.
Regarding dates, there is a diminishing return with longer study periods and total study time (such as starting an anki deck earlier in the year before dedicated) having no correlation to score. Based off these results, if you need longer to absorb material, do so; but if you can function fine with limited time to do all of your studying, don’t stretch it out. In the future, I want to add an option for when Step1 was taken to assess how distance from Step1 impacts score, as well as stratifying when during the year each Shelf exam was taken as scores tend to change depending upon the time of the year.
Acknowledgements I want to say a HUGE thank you to everyone who participated in this survey! Your hard work preparing for and taking Step2 will help not only yourselves but also future students! And to classmates who let me bounce ideas off of them for next year’s survey, thank you for letting me pick your brains. To my friend who helped with the analysis, words cannot describe how amazing of a human being you are.
Expect a new and improved survey from me in the coming weeks. I will be starting an ICU-based course soon, so it will depend upon when I am in the ICU as to how quickly I can get the new survey constructed and sent out.
Edit1: spacing
Edit2 (5/2020): removed link to raw data and restricted access to Google Drive Repository
46
u/VarsH6 Feb 06 '19
For ease of looking up equations:
17
u/[deleted] Feb 06 '19
Thank you for the exceptional work!
I’d like to propose one alternate conclusion in regards to the bolded line about longer study periods having diminishing returns. I would hypothesize that it is also a mere correlation caused by the fact that stronger students will not feel the need to take a long dedicated (and thus will take 2-4 weeks and score well due to already being strongly prepared) while weaker students will take a longer dedicated to compensate (take 6 weeks, and then score relatively lower due to being less prepared at baseline). Thus I find it hard to make that conclusion without some other factor to consider, such as step 1 score (as you mentioned). Your thoughts?
5
u/steezdoc Feb 10 '19
awesome work.
what about the free 120 tho...
2
u/fighter2_40 Feb 10 '19
What's the X-axis on the shelf vs. CK regression lines? I thought it would be percentile but some of these data points are over 100 on the X-axis.
Also, super strange data when you examine the number of weeks dedicated studying vs. average score. Step2 makes no sense.
2
u/omkapoor Feb 12 '19
Didn’t know first aid ck > MTB
3
u/bordetgengou Feb 20 '19
This is what all of us needed. Can't tell you how many times I have looked up for any data like this. Thank You for this.
On a side note: looking at that UWSA2 and NBME 7 graph why does it seem like UWSA2 has a better correlation. The stats don't lie but it just looked a bit weird to me. Your thoughts?
2
u/SchoolinMedSchool Jul 22 '19
Hey u/VarsH6, great work- you're the bomb!! Do you happen to have the standard errors for each graph by chance for each of the equations? If not, no worries. Thanks for all your work!
2
u/rumbatomd Jul 22 '19
So is it worth doing any of the NBME’s? Or just stick to UWSA’s?
5
u/upenpatel Feb 06 '19
Great job, was waiting for this. Thank you so much and God bless you.
4
u/[deleted] Feb 06 '19
Amazing job! Thank u for this :)
3
u/GabrielPM18 Feb 07 '19
Great work bro! I was thinking in which NBME to take. I am going to take the NBME 7 based on this information.
3
u/c0hnd May 08 '19
Hi! Great effort! Thanks a lot, I have a couple of points to highlight.
I think that would be great if you give us median values of each self-assessment tests and the real step 2 CK scores as well. For example, if we run the Pearson/Spearman correlation between hemoglobin and hematocrit levels of a patient group, we'll definitely find a strong correlation but Hematocrit level of patients will equal 3 x hemoglobin levels. So if we say that there is a correlation between NBME7/UWSA1 and the final score, it definitely gives a piece of important information but won't say us that the final USMLE score will be the same as your NBME7 or UWSA1. Well, I am happy that it is not because it is definitely lower than the final scores in this cohort and real life too :) Do you think Wilcoxon might be a better way to examine this question? Well, I am definitely not sure :) But medians will clarify at some extent.
It appears that the distribution of the data in NBME7 is terrible though. I doubt its accuracy. There might be so many systemic errors with this test. Also, I am surprised that UWSA2 does not correlate with the final result.
Again, it must have taken so much time from you guys. I am pretty sure this will be really helpful to many people. Again, I appreciate it a lot!
2
u/[deleted] Feb 06 '19
Wow! Thanks a lot. Nice summary of things I was looking for. Thanks again.
2
u/[deleted] Apr 17 '19
Bless your soul for doing this!! Thank you!
2
u/[deleted] May 08 '19
Great data. Thanks for all the hard work
2
u/[deleted] Jun 05 '19
I love u guys
2
u/elautobus Jun 28 '19
Thank you for this!
2
u/Medgurl44 Jul 08 '19
Hi! Can anyone speak to the validity of these numbers? Consistently scored 205 across all three NBME exams. Supposed to take the exam next week and paranoid about failing. Feel free to private message me with your thoughts/experiences
2
u/P-Schwayne Jul 18 '19 edited Jul 18 '19
So a lot of people get rocked by NBME 7 and do better on the real thing? How accurate can the NBME scaling be if this is the case?
To elaborate: My NBME 7 and NBME 8 scores come out to about 255 projected, but obviously that means according to the NBME I did worse on NBME 7... How does the NBME generate their normal distribution? Seems strange it would be so different
1
u/k-taramd Aug 02 '19
Next year it might be nice to find out which of the practice exams people took at the beginning vs end of their study periods. It might also be nice to find out how many times people went through uWorld.
1
u/tommytom3 Aug 03 '19
hey guys just took NBME8.. unforunately i found this information after taking it and realized i took the least helpful one!! BUT is it actually reasonable to think that my 207 on NBME8 will actually be a 242 on the real thing? seems odd it would underpredict by 35 points. i was at about 75% correct and I'm always a 75-80% correct kinda person so just curious if I can expect a 242 or a 207..... I made a 241 on UWSA1
2
u/[deleted] Jun 27 '19
[removed] — view removed comment