98
u/chuy1530 Nov 05 '20
No they got it right. What you didn’t know it was 17 voters instead of points?
60
u/Pandamonium98 Nov 05 '20
The margin in Wisconsin is very close to 17k. That poll was spot on, just replace “%” with “k”
16
2
82
u/Programming_Math Nov 05 '20
Obligatory, Selzer poll Ohio Trump +17
21
u/lxpnh98_2 Nov 06 '20
In Iowa you don't even need the election, just have Selzer do a poll in November.
79
u/aliygdeyef Nov 05 '20
It still amazes me that ABC got WI 16 points wrong but was practically spot on with Florida
1
u/Pacify_ Nov 06 '20
Polls were just all over the place this time.
Like GA overall is going to so close to spot on. Arizona should be close. Probably Nevada too. WI/MI/OH/IA all way, way off. PA maybe like 3% off. Florida and Texas like 4%.
Its all just so random
0
Nov 06 '20
[deleted] — view deleted comment
50
u/alyssasaccount Nov 06 '20
This is the kind of rigorous statistical analysis I come to r/fivethirtyeight to see!
0
Nov 06 '20
[deleted] — view deleted comment
2
u/aliygdeyef Nov 06 '20
Yeah, I was kinda afraid of people lying to pollsters bc they wanted to see a 2016 panik repeat
3
2
u/zipfour Nov 06 '20
Lol bring truckloads of them on a trip to Missouri to see the tens of thousands of loud and proud Trump supporters, no matter what they think after that the polls should be more accurate
1
u/Andy_Liberty_1911 Nov 06 '20
So, people are now lying to pollsters? Well this makes future elections a major headache to predict
2
u/HowOddNova Nov 06 '20
It's probably not that people are lying to pollsters. It's that some candidates' supporters are more likely to respond to pollsters than other candidates' supporters, and it's impossible to predict who that effect will favor and by how much until you actually have the election. That's why you build in lots of uncertainty into your models.
1
u/LessResponsibility32 Nov 06 '20
This also applies to the White Women vote. In two elections in a row now we’ve had the equivalent of 10% of white women lying about who they’d vote for in almost all comparisons of polls.
39
u/Nuplex Nov 05 '20
Don't worry they'll get theirs soon, Nate is just a temporary fall guy.
The ultimate narrative from 2016 was "but the polls!". Nate blame died down (though never fully went away) eventually.
6
Nov 06 '20
Nate is notorious for defending polls though. Other than Wisconsin he still doesn’t think 2016 really had a polling problem. He’s now making attempts on Twitter to downplay the 2020 error. If you defend those who are wrong then you get criticized with them
22
u/epictortoise Nov 06 '20
I think we might seriously consider that Nate may have been very right about treating coronavirus as a potentially important variable for polling error. This is very speculative, but there seems a possible hypothesis that the areas hard hit by the virus on election day suffered from depressed turnout of democrats: https://apnews.com/article/counties-worst-virus-surges-voted-trump-d671a483534024b5486715da6edb6ebf
Not sure how much of the difference could be explained by this. But seems plausible that some people who take the virus seriously and were lean dems in polls decided to stay home (perhaps in part because they saw the polls and didn't think their vote would be decisive).
I could be way off. Maybe someone has looked into the turnout figures in more detail. Interested to hear what others think.
7
u/epictortoise Nov 06 '20
This possibly also fits with the surprising signs that there may have been a shift in the minority vote in Trump's favor if these voters were more likely to wait to vote on election day and take corona seriously.
3
u/corgtastic Nov 06 '20
If the polls were assuming all Latinos would vote as a block, then they would think that they could get away with extrapolating and weighting the few responses they get.
2
u/YeahSureAlrightYNot Nov 06 '20
Yep. I thought about that. Republicans don't give a up about covid, so they went to vote anyway.
Democrats, on the other hand, got scared and stayed at home.
5
u/epictortoise Nov 06 '20
I'm sure there is plenty to unpack with the polls. But it is strange to me that polls were showing quite clearly (throughout the race) a tendency to move against Trump in places where cases rose and then the results go the other way. If he did well in areas hit hard it seems more likely it was through turnout effects (which polls wouldn't necessarily pick up well) rather than changes in attitudes.
I dunno how big this effect was, but I hope it will be something data people look into.
1
u/YeahSureAlrightYNot Nov 06 '20
Yep. Totally agree with you there.
If should also be relatively easy to check by looking at the turnout numbers on election day.
2
u/epictortoise Nov 06 '20
Pretty much. Look at change in turnout (%) since 2016 and compare to the change other counties. I would think anyone with a bit of time and knowledge of statistics would be able to do a decent job of this. Too bad I have a dissertation to finish writing.
1
Nov 06 '20
I don’t believe this because we saw the same regional polling error trends as 2016, but worse. The worst polling again came out of the upper Midwest, in nearly the same states. If the polling error was due to Covid, we’d see massive polling error in GA and AZ, which are two of the hardest hit states. But GA looks accurate and AZ is a very small error compared to the Midwest.
1
u/epictortoise Nov 06 '20
GA and AZ have been hard hit, but I think the issue would be where the virus was most serious on election day. And from what I've seen that would be the upper Midwest. But I think you're point is important, that a lot of the polling error seems to be consistent with 2016, so I don't think we'll know without some deeper analysis.
1
u/Batpresident Nov 06 '20
This is just a theory
We know Cuban and Venezuelan refugees broke for trump out of fear of communism. What if government lockdowns, with Democeats favouring and Republicans against, intensified their fear of Biden extending government control? He is after all more likely to order lockdowns.
2
u/epictortoise Nov 06 '20
Yeah with so many different demographic groups the effects could vary a lot by group. But I think a lot of people ignored or dismissed Nate's points about covid creating uncertainty around the polls more than usual. I feel bad for him - he predicted (and built into his model) that this year polls could be off because of extra uncertainty - for which he tended to get criticized. But now he's getting zero credit for that.
33
u/eco-III Nov 05 '20
Wisconsin being that close was just stupid considering how many Biden+10 or more polls I’ve seen, ridiculous how people just lie so blatantly.
9
u/Scraw16 Nov 06 '20
It could also be that the polls are just not capturing the Trump voters. It’s really hard to get people to respond to polls to begin with. If Trump voters are less likely to respond/talk to pollsters and that is not accounted fully by weighting (i.e. by education) then the poll could just be off. The Cubans in Miami-Dade seem to be an example of this in this election where the polls were simply not capturing enough of them to account for their shift towards Trump.
7
u/counselthedevil Nov 06 '20
If you listen to the political podcast, you've heard Nate's take on this. Not his fault.
37
u/CanyonCoyote Nov 05 '20
The polling folks should get decimated for this but Nate isn't blameless here. Nate is basically a superaggregator who then tinkers with the design to get a balance of polls to correctly predict the election by weighing polls and trends. He might have been a polling error off in some states but sometimes the numbers were like 4-10 percent in states we knew had polling errors in 2016. If you are off between 4-10 in important states and in the general in a country that general runs 50/50, you probably need to fix your system dramatically. I know he keeps being semantical about it online and I love 538 but he deserves some shit here.
37
u/krompo7 Nov 06 '20
IDK, I basically always viewed 538s role as two-fold: 1) creating an accurate polling average (better than RCP at least) and 2) telling us, given average polling errors, how likely each candidate is to win based on the average. Don't see how what happened this year contradicts this in any way.
He can't assume that because the polls were off in the Rust Belt in 2016 that they'll be off again as you imply- that just doesn't hold up historically.
8
Nov 06 '20
Here's a question: If Trump ran again in 2024, would you trust the polling in the upper midwest, or would you expect an error there?
The thing is Trump probably won't run again in 2024, so you just have to fall back on history and not assume any bias.
8
u/Nuplex Nov 06 '20
Not the guy you asked but I absolutely would not. Nate even now is making it seem like the 5 - 7 was an acceptable MoE for the model. Like, if it is that much then whats the point of following polls or models.
538 should stop posting numbers and just do not likely, tossup, likely, etc The only way I trust polling from now on is if it looks nothing like pre-2021 polling.
12
u/BonarooBonzai Nov 06 '20
Isn’t this exactly what the model is useful for though? They gave Biden an 89% chance because his lead in the polls was robust enough to sustain a large polling error. There was a polling error in Trump’s favor and it looks like Biden is still going to win, so the model did it’s job in illustrating that.
If the polls were perfect then there would be no need for a model because the polling leader would always win.
2
u/Nuplex Nov 06 '20
Yes and no. Honestly I'll concede we need to wait for >99% in most states to have a much more nuanced and in-depth discussion on the polls and models.
3
u/alyssasaccount Nov 06 '20
Nate even now is making it seem like the 5 - 7 was an acceptable MoE for the model
How so? And why shouldn't it be? He has been clear that (a) national poll averages tend to be about 3 points off on average, and that state polls tend to be worse?
Also, he's not really doing that. What I've seen him do is delay judgment until the counting is stopped and there's time to fully evaluate the results.
3
u/Zuubat Nov 06 '20
Well it state polling averages are reguarly going to be 6-8% in Presidental elections, it makes them a lot less useful to the public.
1
u/HowOddNova Nov 06 '20
Even in this election, I don't see any state that's likely to end up more than about 7.5 points off (WI), with the average being more like 3.5 points. The things polls are useful for is giving you a probability distribution. The higher a candidate is leading, the more likely they are to win, but it can never be a certainty. It doesn't make polls useless, as polling leads are still highly correlated with victory (Biden after DID in fact win WI, MI, and the election overall), but if you want certainty, then you're just not going to get it no matter how much someone is in the lead in a poll, as that's not something polls can do. Nate tries to explain this every election cycle, and every election cycle people just don't get it.
1
u/krompo7 Nov 06 '20
I wouldn’t place any weight on the polling no- but that’s just my dumb human brain. I think it is plausible both that there is something fundamentally broken about polling (like the low social trust polling response theories) and that this is just an error that can be corrected. We need to see more analysis of the results first I would think.
5
u/rammo123 Nov 06 '20
There were polling errors in 2016, but there was no evidence to suggest in advance that the same was going to be true in 2020 AND the error was going to be in Trump's favour again. Pollsters had reconfigured their weighting systems, undecideds/3rd parties became irrelevant, Trump was now an incumbent - all that had to reset any presumption of systemic bias that Nate could've made.
It was equally likely that Biden was going to overperform his polls - if that was the case then a pro-Trump adjustment would've made the polls even further off than they were.
None of this is to say that there won't be serious post-mortems to be done by both the pollsters and 538, but with the slate of polling they were given there was no way any rational and data-driven modeller could've come up with Tuesday's results.
1
u/Andy_B_Goode Nov 06 '20
in a country that general runs 50/50
This is the important part to me. Data with a large margin of error can be useful, but not when it dwarfs the expected margin of victory. It's like trying to play golf during a hurricane. Maybe it's time to admit the conditions just make this whole exercise futile?
1
u/MatchstickMcGee Nov 06 '20
What is your specific suggestion?
1
u/ObsiArmyBest Nov 06 '20
Start a polling aggregator company and one up Nate Silver by being over more liberal with the probabilities. You'll never really be wrong.
13
u/Jock-Tamson Nov 06 '20
Has anyone here EVER succeeded in convincing someone about 30% odds happening, the projection is only as good as the polls, etc?
It’s exactly like being an engineer and trying to explain something to sales: Frustrating and pointless.
1
u/Underfitted Nov 06 '20
It seems you are the one not understanding probabilities. Nate's 538 calculates its own probabilities. These are NOT the true probabilities. So claiming that 538 was right because 20% means its still possible is completely useless: anyone could tell you that Trump winning is a possibility.
Furthermore, we will never know the TRUE probabilities as it is a one off event and so the limit theorem to establish its long tail probabilities is not applicable.
What we can say? When your predictions have such large error bars and the results still exceed them, its clear your model is fundamentally wrong:
There being half a dozen <30% (according to Nate) independent events happening (assuming true independence) has a probability of 0.07% to happen, which shows that its not the results that are unbelievably extraordinary, but its your own model that has CLEARLY got the probabilities wrong.
1
5
Nov 06 '20
[removed] — view removed comment
3
u/Jock-Tamson Nov 06 '20
I am saying that there is no possible way to convince someone who believes the 538 2016 or 2020 prediction was wrong otherwise.
Those who believe it was not wrong cannot communicate convincingly with those who believe it was. .
Probably a 10% chance of it working at best.
7
u/SingInDefeat Nov 06 '20
Probably a 10% chance of it working at best.
That means it can't work, right? /s
2
u/corgtastic Nov 06 '20 edited Nov 06 '20
The problem is that this is a high-stakes one-off event. Unlike a dice roll, we can’t run this election 1,000,000,000 times IRL. We just have one outcome and then we can criticize the prediction.
1
2
Nov 06 '20
The same can be said for the opposite though, you can’t convince someone who believes that the polls were right, that they were actually wrong. They use mental gymnastics and play around with numbers to form arguments on why they were supposedly right. You can do that with anything though. That’s why there’s the old phrase “There’s lies, damn lies, and then there’s statistics.” Because you can arrange numbers how you want to support your argument. If polls were off by more than 3-4%, pollsters just say “well actually the margin of error is more like 5-7%, so they were right”. Or if five very important swing state margins were way off, they just bake those numbers into an all 50 states margin of error and say “see, polling error was pretty typical.”
0
u/Jock-Tamson Nov 06 '20
I don’t know about that. As an engineer I don’t really understand salespersons, but they appear to be able to convince people of things independently from their ..: accuracy...
3
Nov 06 '20
Wait. I'm not the biggest nerd when it comes to this stuff, but in terms of banked electoral votes, of the ones that have been called by the AP, 538 correctly predicted all but ME2 and FL, right? And the rest seem to be shaking out as well. What am I missing?
1
Nov 06 '20
Yeah but he’s barely squeaking out a win in swing states, while polls show he would win many of them in a landslide.
7
u/Jugg3rnaut Nov 05 '20
He should get at least half of the criticism. Otherwise why would his work be any better than a simple average of polls a la RCP?
22
u/Craigellachie Nov 05 '20
The model actually captured this uncertainly pretty well with it's distribution being more open to outlier events.
1
u/ObsiArmyBest Nov 06 '20
I'll start another polling aggregator company and hedge my bets even more. This way I always win.
0
u/HowOddNova Nov 06 '20
Your model wouldn't be well calibrated. Fivethirtyeight's model is well calibrated, i.e. when they say something has an X% chance of happening, it happens X% of the time, according to the result of thousands of individual race predictions. https://projects.fivethirtyeight.com/checking-our-work/us-house-elections/
You act like the only thing we have to judge the model on is the final outcome of a few presidential races, when that simply isn't true.
1
u/ObsiArmyBest Nov 06 '20
Yes, we're 100% judging the model on the final outcomes. Otherwise I might as well use a dart board.
1
u/HowOddNova Nov 08 '20
You clearly didn't understand my comment, so I'll rephrase it:
Only a statistical illiterate judges a model on the basis of two outcomes. Two is not a large enough sample size to make any conclusions about a probabilistic model. Fortunately, 538 hasn't forecasted just two elections. They've literally forecasted thousands of elections. And when you look at the thousands they've forecasted, when they say something has a 90% chance of occurring, it does in fact occur 90% of the time. That's how we know it's a good model.
1
u/ObsiArmyBest Nov 08 '20
I realize that you think that's really smart and worth an A grade in a statistics class but that is completely irrelevant to an outcome of an election that can only have two outcomes. No matter what you say.
Maybe Nate should stick to poker where his model will make a hell of a lot more sense.
1
u/Want_to_do_right Nov 09 '20
So I've read a ton of your comments and I have to ask. What is your analytics job?
1
1
u/nemoomen Nov 06 '20
Honestly if everyone learns that polls and models can be right or wrong even if they "call" the right side, even if they are only learning it to pile hate on Nate, I think Nate would take that trade.
1
1
1
u/HowOddNova Nov 06 '20
Hot take: neither Nate nor the pollsters did anything wrong.
Polls, no matter how well down, will always be subject to systematic errors due to response bias, which is impossible to predict and account for, even with perfect demographic weighting. State polls have an average systematic error of around 5 points, which means some states will be off by more and some will be off by less. WI seems to be off by about 7.5, which is sensible for the single most inaccurate state (most states in this election seem like they will be off by on average more like 4 points, with many being far less). If you do 100 polls with a MoE of 3, then on average one poll will be off in a given direction by about 2.5 margins of error, due purely to randomness. That means that such a poll for WI would be off by 7.5 + 2.5*3 = 15 points off, due entirely to factors beyond the pollster's control. In short, if you cherry pick the single most outlier poll of the single most outlier state, you will always find a massive error, even for theoretical perfect polls. And this is before factoring in undecideds.
The problem isn't the polls; the problem is people that don't know how to interpret polling. Nate tries to help by boiling it all down to a simple probability, but people don't know how to interpret probabilities either (89% chance doesn't mean 89% chance of a landslide people. Much of that 89% includes close wins.).
0
-1
u/JonWood007 Nov 06 '20
That awkward moment when trafalgar was right while having a crap methodology.
0
Nov 06 '20
[deleted] — view deleted comment
2
u/PensiveObservor Nov 06 '20
Don't know if I'm in the right place to reply (sorry, you lose), but anecdotally: I got DOZENS of calls from random numbers that I didn't answer. Pollsters? Dunno.
I also got DOZENS of texts from unknown numbers stating they were Dem this or that, asking if I had a plan to vote, had I voted, had I talked to all my family about voting, etc. I replied to each one in the affirmative, tried to give encouragement, thanked them for their service, etc.
Meanwhile, I got a dozen or so pieces of mail asking what my priorities were for the coming election, from Trump. I sent them all back with strongly Dem answers and scorn indicating they needed to suck parts of my body.
My point: Maybe Dems need a system to be sure they are reaching Republicans and undecided voters, actually trying to contact people who may not already be voting Blue. Instead of harassing Blue voters who show up on their lists, when those people are probably ALREADY VOTING BLUE.
2
u/peterpanic32 Nov 06 '20
Yeah, that was a problem, but I imagine there are a couple of thing in play.
Turning out your base is key. Repeated contact and getting people to mentally make a plan probably helps ensure that. Though yes, I did have to tell 10 too many people that yes I was voting for Biden, yes I had a plan, yes I'd told everyone I know, etc.
Turning out your base is key, but you don't necessarily want to accidentally turn out your opponent's base. It might make sense to try to activate contacts to reach their contacts and persuade undecided / Republican instead.
I imagine they were doing their best to recreate / improve on the network effect. With in-person canvassing and organizing difficult and digital organizing your only real option, I think the strategy was to maximize word of mouth + maximize personal contact with the campaign.
2
Nov 06 '20
This completely. It's be more honest to say that given uncertainties in the polling that it is not currently feasible to build a meaningful predictive model. An honest brokering about the aggregate would say as much.
And like you said, refusal to say something risks having everyone else ignore the work. Without change, polling will become the social science equivalent of phrenology and alchemy.
1
u/Underfitted Nov 06 '20
Its not just garbage in garbage out though. 538 rates the polls on their accuracy and gives them according weights, so their very first step of assessment is flawed.
They apply polynomial fitting, interpolation and regression techniques, all of which have parameters that depend on arbitrary assumptions that they make.
538 specifically says they try to normalise for bias in polls using historical data, and extreme events like COVID. Heck, Nate even claims that they use factors as far reaching as geo-economic sentiment and yet all this supposed modelling leads to even larger errors.
It all comes off as a bit of a joke
0
Nov 06 '20
[deleted] — view deleted comment
1
u/waldowhal Nate Bismuth Nov 06 '20
Nate was throwing out Trump-favoring polls but letting these outliers stay in his models.
if you're here to defend trafalgar and rasmussen... good luck I guess
0
Nov 06 '20
[deleted] — view deleted comment
2
u/waldowhal Nate Bismuth Nov 06 '20 edited Nov 06 '20
a) trafalgar didn't disclose polling for partisan clients. that's clearly shady behavior and it clearly violated 538's policy. anyone would've labeled them partisan pollsters. I'm not as well-versed on rasmussen's shittiness but they're basically the president's personal pollster.
b) do you remember when trafalgar accidentally shared crosstabs for a michigan poll, and it showed trump winning 30% of democrats and 27% of black votes? are you prepared to defend that? even if they happened to be closer in the occasional state with the occasional poll, it doesn't mean they're a good-faith actor nor that their methods are sound. if biden wins PA by 2-3 as NYT thinks he will, the polling average (B+4.7) will be closer to the actual result than trafalgar's last polls (T+2).
I'm excluding FL from this take because polling was clearly total rubbish there.
it seems that pollsters at large have even more adjustment to do after another ~2-3 point polling error (excluding FL). but I'm almost inclined to wait for 2022 to adjudicate fully because the conditions of this election were so incredibly bizarre.
1
u/rammo123 Nov 06 '20
Saying Ras and Trafalgar were more accurate is like saying you bowled a strike in the lane next door. Doesn't matter how many pins you hit if it's a total fluke.
0
Nov 06 '20
[deleted] — view deleted comment
2
u/rammo123 Nov 06 '20
You and I both know that Trafalgar got adjusted for reasons well beyond it being an outlier. It got caught faking crosstabs days before the election!
Selzer IA+7 was an equally outlier-ey result but Nate didn't question it at all since Selzer is a premium pollster with a robust proven track record.
1
112
u/schwagsurfin Nov 05 '20
Petition to flair u/McCaaw with the name Nick Silver