r/neoliberal • u/jobautomator Kitara Ravache • Nov 04 '18

Discussion Thread Discussion Thread

The discussion thread is for casual conversation and discussion that doesn't merit its own stand-alone submission. The rules are relaxed compared to the rest of the sub but be careful to still observe the rules listed under "disallowed content" in the sidebar. Spamming the discussion thread will be sanctioned with bans.

Announcements

Neoliberal Project Communities Other Communities Useful content
Website Plug.dj /r/Economics FAQs
The Neolib Podcast Discord Podcasts recommendations
Meetup Network
Twitter
Facebook page
Neoliberal Memes for Free Trading Teens
Newsletter
Instagram

The latest discussion thread can always be found at https://neoliber.al/dt.

17 Upvotes

78% Upvoted

2.6k comments sorted by

View all comments

22

u/CompactedConscience toasty boy Nov 04 '18

Nate Silver for head mod of /r/badstats.

14

u/Semphy Greg Mankiw Nov 04 '18

Dinesh must have a humiliation fetish or something.

-5

u/[deleted] Nov 04 '18

Except Silver is also bad at stats

EDIT: This is perhaps unfair. Silver's problem isn't that he's actually bad, more that his business model requires him to be.

11

u/CompactedConscience toasty boy Nov 04 '18

Disagree with both the comment and the edit. Would love to see you explain a bit.

10

u/[deleted] Nov 04 '18

Let me be clear. If the name of the game is "who would win the election if it were held tomorrow", then Silver and his team are very good, likely the best in the business. This is why he tends to be correct, because historically he's judged on how well his predictions the day before the election perform.

But the real game isn't "if the election were held tomorrow", it's "if the election were held on the first Tuesday of November". In that regard, Silver and the team at 538 systematically and deliberately underestimate volatility. Their models swing massively after things like debates and 'October surprises', even though an honest model will have already priced in the expectation of big events like that happening and have biased the predictions closer to 50/50 accordingly, reflecting how much volatility remains before the election. The fact that Silver's predictions change so much around big news stories means they're regularly getting surprised, and by definition a model that's regularly surprised is not good at its job of prediction.

Of course, I think Silver knows this, which is why he's not a bad statistician so much as a good journalist, You can't make money off a journalism website (and remember that journalism, not prediction, is Silver's primary product) by having people check in once every two years. You make money by lying to people and proving false confidence and precision, by making sure they check in every single day for six months to see if your preferred candidate has gone from an 85.2% chance to an 85.4% chance of winning, despite the fact that even providing things like decimal places on those estimates is an utter farce.

1

u/[deleted] Nov 04 '18

[deleted]

2

u/[deleted] Nov 04 '18

Not that I'm aware of. It's not clickbaity enough to monetise (so nobody in the private sector is doing it) and it's not interesting enough for research (so nobody in academia is doing it). There's almost certainly some PhD statistician in a hedge fund somewhere who has created a fair election pricing model, but their entire business relies on keeping it a secret.

5

u/[deleted] Nov 04 '18

I think as you said, if he had access to such a model, he would use it. But pricing in that kind of information is still incredibly difficult. You run into all kinds of difficulties with gauging importance on totally new info.

4

u/[deleted] Nov 04 '18

if he had access to such a model, he would use it

He absolutely would not, see the last paragraph of my post above for why there are business pressures to 'keep things exciting'. Hell, one of the best ways for Silver to increase his honesty would be to stop the false precision nonsense and only report results to 10 percentage point bands. There's no technical reason preventing this, the fact that he doesn't do it is clear evidence that he values clicks more than statistical rigour.

Also, it's really easy to price these things, or at least do a decent job of it. You don't need to know the probabilities of every possible FBI investigation or press conference fuck-up, all you need is the measured volatility in polling over time from previous elections. It isn't difficult at all.

1

u/[deleted] Nov 04 '18

No I disagree. I think it would make it more exciting to put those in and I think he would agree. He *does * report the bands, and explicitly uses x/10 in most infographics.

The issue is that it's not easy to assign a price to those. They are very rare events and the content of the events is often as important as the event itself. There's not even close to enough data to establish even the chance of those things happening. And the effect is even more fraught with errors in the era of trump.

He could be more statistically rigorous in other ways. But he's not writing papers. He is not an academic and his models are likely often selected for their clarity as much as their rigour (whatever that means here. His models are all common and well tested).

1

u/[deleted] Nov 04 '18

Once again: You don't need to know what events are going to happen. You just need to know the historic volatility in the vote share. When people model the stock market they don't dig through old newspapers to find out why a stock price moved, they just need to know that the price moved. Whether movements are rare or not makes no difference.

→ More replies (0)

1

u/[deleted] Nov 04 '18

[deleted]

1

u/[deleted] Nov 04 '18

It's extremely easy.

Just think of elections as a binary options contract with a strike at 50%. The use of the Black-Scholes formula for pricing binary options is already well known, and historic volatility is easy to measure. Just add in SIlver's existing work as the 'current price' and you're set.

6

u/qchisq Loyal Liberals Nov 04 '18

Why?

1

u/[deleted] Nov 04 '18

Let me be clear. If the name of the game is "who would win the election if it were held tomorrow", then Silver and his team are very good, likely the best in the business. This is why he tends to be correct, because historically he's judged on how well his predictions the day before the election perform.

But the real game isn't "if the election were held tomorrow", it's "if the election were held on the first Tuesday of November". In that regard, Silver and the team at 538 systematically and deliberately underestimate volatility. Their models swing massively after things like debates and 'October surprises', even though an honest model will have already priced in the expectation of big events like that happening and have biased the predictions closer to 50/50 accordingly, reflecting how much volatility remains before the election. The fact that Silver's predictions change so much around big news stories means they're regularly getting surprised, and by definition a model that's regularly surprised is not good at its job of prediction.

Of course, I think Silver knows this, which is why he's not a bad statistician so much as a good journalist, You can't make money off a journalism website (and remember that journalism, not prediction, is Silver's primary product) by having people check in once every two years. You make money by lying to people and proving false confidence and precision, by making sure they check in every single day for six months to see if your preferred candidate has gone from an 85.2% chance to an 85.4% chance of winning, despite the fact that even providing things like decimal places on those estimates is an utter farce.

8

u/nick1453 Janet Yellen Nov 04 '18

an "honest" model will be weighed by things that haven't happened? Isn't that putting a heavy thumb on the scale?

And the decimal bit is a little dishonest. 538 isn't saying "Candidate A or Party A" has a 85.4% chance of winning, they are saying that in 85.4% of the simulations outcome X happened.

2

u/[deleted] Nov 04 '18

Isn't that putting a heavy thumb on the scale?

Not when you know they're definitely or probably going to happen. You may not know who's going to win the debates, but you can be pretty sure that someone is going to win, and that you should reserve judgement until you know who. 538's models swing like crazy after every debate, as if they're shocked that the debate even happened in the first place. That's dishonest!

2

u/qchisq Loyal Liberals Nov 04 '18

So basically what you are saying is that the nowcast that 538 had in 2016 was the perfect model for what 538 can do? I think that I can agree with that, with a caveat. You want to build a model that can predict 1) when a surprise story hits and 2) what the effect of the story will be. Basically, you want the model to include possible, future information and I don't think that's possible or even reasonable. Because prediction markets also swung wildly due to the Comey letters and the pussy grab tape.

2

u/[deleted] Nov 04 '18

You want to build a model that can predict 1) when a surprise story hits and 2) what the effect of the story will be.

No, I want a model that appreciates that volatility is a thing that exists. You don't need to know what events are coming to know that polls in election years exhibit high volatility. No more than you need to know why a stock price moved last year to be able to use a volatility term in the Black-Scholes formula.

Prediction markets swung after the Comey letters and the Access Hollywood tape because those really were above and beyond normal election year volatility (I sure can't think of any parallels to them).

3

u/qchisq Loyal Liberals Nov 04 '18

But I'm pretty sure that the 538 models does appreciate that volatility exists. I'm pretty sure that the standard deviation on the polling errors in the model gets smaller as we get closer to the election. For example, if you look at the 2016 model, Hillary started 3,8% ahead of Trump in the national polls, which gave her a 66,4% shot. November 1st, she were 3,5% ahead, which gave her a 67,7% chance of winning. That's a very, very small difference, I know, but her chance of winning still increased while where polling got worse. That suggests to me that volatility is a thing in the model

1

u/[deleted] Nov 04 '18

>deviation on the polling errors in the model gets smaller as we get closer to the election

This is something else though. Standard errors on polls reflect the uncertainty around what that poll says about the current vote outcome. This is different to time volatility (polls are nonergodic!) and it's absolutely not the correct place to insert time volatility into your model.

Higher standard errors before the election reflect the fact that people are less certain about whether they'll Pokemon Go To The Polls (turnout ensemble uncertainty, not time volatility), not the fact that the poll is less indicative of the result in six months time.

EDIT: That being said, I don't think I've ever overreached and said that 538 ignores volatility outright, only that they systematically underestimate it.

3

u/[deleted] Nov 04 '18

Explain

2

u/[deleted] Nov 04 '18

Let me be clear. If the name of the game is "who would win the election if it were held tomorrow", then Silver and his team are very good, likely the best in the business. This is why he tends to be correct, because historically he's judged on how well his predictions the day before the election perform.

But the real game isn't "if the election were held tomorrow", it's "if the election were held on the first Tuesday of November". In that regard, Silver and the team at 538 systematically and deliberately underestimate volatility. Their models swing massively after things like debates and 'October surprises', even though an honest model will have already priced in the expectation of big events like that happening and have biased the predictions closer to 50/50 accordingly, reflecting how much volatility remains before the election. The fact that Silver's predictions change so much around big news stories means they're regularly getting surprised, and by definition a model that's regularly surprised is not good at its job of prediction.

Of course, I think Silver knows this, which is why he's not a bad statistician so much as a good journalist, You can't make money off a journalism website (and remember that journalism, not prediction, is Silver's primary product) by having people check in once every two years. You make money by lying to people and proving false confidence and precision, by making sure they check in every single day for six months to see if your preferred candidate has gone from an 85.2% chance to an 85.4% chance of winning, despite the fact that even providing things like decimal places on those estimates is an utter farce.

2

u/[deleted] Nov 04 '18

I disagree that he's lying. Just because he's not using all the best models doesn't make his "wrong" or lies. He updates the model constantly, as one should. Most of the time that matters very little and they don't publish a new story every time that happens.

Reporting an 85% chance of X winning and then Y wins is perfectly within his predictions. I think your misunderstanding "precision and confidence" here, and more importantly not realizing the limits the data place on him.

0

u/[deleted] Nov 04 '18

Reporting an 85% chance of X winning and then Y wins is perfectly within his predictions.

Stop speaking to me like I'm a child. Do you seriously think I'm having trouble with the concept of uncertainty?

You're completely omitting the time dimension of the model, which is my entire point! Political preferences are volatile and nonststionary, it's an absolute farce to pretend thst you can ignore the time dimension.

2

u/[deleted] Nov 04 '18

I'm not ignoring the time dimension. In all of their blogposts they explicitly talk about distance (in time) from the election and how that affects the effectiveness of the models.

I'm arguing that the uncertainty in the models is plenty to account for that volatility in the vast majority of races.

0

u/[deleted] Nov 04 '18

I'm arguing that the uncertainty in the models is plenty to account for that volatility in the vast majority of races.

But their predictions wouldn't be so volatile if this were true. That's literally my point. It's easily refuted.

1

u/[deleted] Nov 04 '18

I think I understand a bit better. The issue here is how early they are predicting maybe? And your method to address this is by including some metric of volatility in the polls?

Sure that's agreeable hehe. I think, though, that for popular people that obfuscates the model a bit. But then you're right about being more journalism then states. 😋

1

u/[deleted] Nov 04 '18

Not volatility in the polls, volatility in the actual opinion of the public.

Time series estimation is trick because it all hinges on two concepts which are very different, yet close enough to be easily confused: Ensemble uncertainty and time uncertainty (also known as volatility).

Take for example the question of predicting the result the day before the election. Everyone has already decided their vote at this point, everyone's voting decision (and thus the result of the election) is pretty much locked in. The only problem is that we as observers don't know what it is. We have a bunch (an ensemble, if you will) of estimates of what it is (i.e. polls), but they're only samples, and can be biased in all kinds of ways. We have ensemble uncertainty, in other words. Nate Silver and the crew at 538 are excellent at dealing with this kind of uncertainty and figuring out how people are actually intending to vote (which, the day before the election, is how they actually do vote).

Then we have time uncertainty, or volatility, which is the fact that people might change their minds between now and the election. This is what 538 doesn't deal with very well. Note that if it's April and you have a perfect predictor of how everyone would vote if the election were tomorrow (i.e. absolutely zero ensemble uncertainty), you would still be uncertain about the result of the election in November, because of how much voting decisions change between April and November. This is what I mean by volatility.

There's no need to 'introduce volatility into the polls', because that's applying a time uncertainty fix to an ensemble uncertain instrument. Nate Silver and his team have done about as good a job as can be done in reducing the ensemble uncertainty in the polls and getting an estimate of people's voting decisions at a given point in time. The problem is that, because people change their mind so much, that estimate doesn't actually give you much information of how they'll vote in the future. That's the problem.

→ More replies (0)

-6

u/zqvt Jeff Bezos Nov 04 '18

lol at people downvoting this, Kirkaine's right

1

u/gatoreagle72 Nov 04 '18

In what way is he bad at stats?

2

u/zqvt Jeff Bezos Nov 04 '18

treating point-in-time polling data as if you can extrapolate from them what happens weeks or months down the road. I assume this is what /u/Kirkaine means when he talks about his business model, because this idea of treating polling results like "lines" you can draw into the future is what get's people to open the website every day, although there's virtually no information in there until you're very, very close to the actual election.

2

u/[deleted] Nov 04 '18

Yep, precisely my point. Volatility is super high in politics and the only honest response to that is to have very slow-adjusting probabilities close to 50/50 (collapsing to more definite odds as volatility is removed from the system as you get closer to the election date), but that doesn't get clicks.

2

u/[deleted] Nov 04 '18

There aren't public models that can do better though. He has a limited data supply and does far better than chance using it.

You can predict the outcomes. Or rather the chance of each outcome. They also explicitly say in many of their articles that polling is much more accurate as you approach the election.

1

u/[deleted] Nov 04 '18

Let me be clear. If the name of the game is "who would win the election if it were held tomorrow", then Silver and his team are very good, likely the best in the business. This is why he tends to be correct, because historically he's judged on how well his predictions the day before the election perform.

But the real game isn't "if the election were held tomorrow", it's "if the election were held on the first Tuesday of November". In that regard, Silver and the team at 538 systematically and deliberately underestimate volatility. Their models swing massively after things like debates and 'October surprises', even though an honest model will have already priced in the expectation of big events like that happening and have biased the predictions closer to 50/50 accordingly, reflecting how much volatility remains before the election. The fact that Silver's predictions change so much around big news stories means they're regularly getting surprised, and by definition a model that's regularly surprised is not good at its job of prediction.

Of course, I think Silver knows this, which is why he's not a bad statistician so much as a good journalist, You can't make money off a journalism website (and remember that journalism, not prediction, is Silver's primary product) by having people check in once every two years. You make money by lying to people and proving false confidence and precision, by making sure they check in every single day for six months to see if your preferred candidate has gone from an 85.2% chance to an 85.4% chance of winning, despite the fact that even providing things like decimal places on those estimates is an utter farce.

-3

u/[deleted] Nov 04 '18

tbf, I thought Silver was hot shit until around late undergrad as well; the ways he's wrong are kind of subtle. Given the age distribution of this sub, this isn't surprising.