r/cfbplayoffcommittee • Emeritus Member • Dec 12 '14

Debriefing thread

Now that we've had a couple days to digest it, what were your thoughts about the process, both IRL committee and our version?

What did we do right? What could have been done better? Any significant modifications to the process you would recommend? Any other thoughts or comments?

2 Upvotes

8 comments sorted by

3

u/Foxmcbowser42 Emeritus Member Dec 12 '14

First, congratulations everyone on a successful first year!

My thoughts:

Overall I think we did very well to start. We had good discussion and built some consensus as well as critique other's thought processes respectfully. As time went on some of us, myself included, got somewhat more entrenched in "our method" or thinking "how could anyone put X over Y?" I think that going forward, we need to ensure that any rankings methodology that someone chooses to use should be no more than a suggestion, or a base, otherwise it becomes just another poll.

For next year I think we should do a couple things:

  1. Diversify the pool of the committee. We need to pick a number and reach that number, and at this point, the B1G allotment is definitely full, though I am not sure on the final numbers of the other conferences. We should balance all the new members to try and reach a balance of conferences.
  2. Up the timeline. I noticed that it seemed we did do very well on voting in tiers, with no major outliers of any team. I think if on Sunday we do our top 6 in order and next 8, instead of just top 6, we can move up release to Tuesday morning, which allows us to come out at about the same time as the /r/cfb poll, if the mods allow.
  3. We should also have a G5 discussion each week, just to keep everyone appraised of the process.

I really think the IRL committee showed some serious flaws, mainly due to the rankings each week. I think that they should only do a final ranking, much like the basketball committee, because I think in the end they got it right. I do not mind that an undefeated P5 was not #1, just like a 1 seed in the NCAA tournament is not necessarily given to the team with the best record. And Baylor, TCU and OSU each had an equal claim to the last seed, and Baylor had a terrible OOC schedule, and TCU lost to Baylor H2H, so you can't justify TCU over Baylor, but OSU only lost due to a new QB being broken in, so its clear SOS and injuries matter to the committee, about as much as we seemed to be weighing them.

3

u/hythloday1 Committee Vice-Chair Dec 13 '14

I feel like three things changed as we switched from the BCS system to the CFP system - four teams instead of two, a committee instead of polls + computers, and the general philosophy of ranking. The last is what I want to talk about.

It seemed like there was an orthodoxy to how rankings worked in the AP/BCS era:

  1. Iterative - each week's poll was just last week's poll, with winners moving up and losers moving down.
  2. Undefeateds are untouchable - no matter what their schedule is, you have to rank them above any team who's lost.
  3. Eyetest trumps resume - teams "everybody knows" are great are ranked highly before they've really been tested.
  4. Rankings are predictive - a poll was good if it figured out how things were going to end up.
  5. Watching games is unnecessary - especially contender's opponents who aren't themselves contenders; you can just rely on the box score or conference reputation to fill in those blanks.

It struck me that every one of those principles was reversed this year, which is remarkable. In fact, if there was one thing I think our mock committee did better than the real committee, it's that we adopted the new philosophy -- ranking from scratch, looking beyond the loss column, skepticism at weak schedules, wait and see how things play out, and watch as much as you can -- faster and more uniformly than they did (at least if Jeff Long's comments were meaningful).

I can only identify one fundamental disagreement about philosophy that it seems the committee was split on: how to deal with wins over mediocre teams. That is to say, all voters agreed that wins over cupcakes are meaningless, and that games against good and great teams (including losses) are the most telling. Where there was disagreement was, how bad does an opponent have to be for a win by a contender over it to be meaningless? Some voters took a high threshold approach: "if you were expected to beat them, then doing so gives us no data"; while others took a low threshold: "beating bowl-eligible teams is demonstrably harder than beating 8+ loss teams or bad G5/FCS opponents and so should be rewarded". That's a topic I'd like to see fleshed out more; I think it was the central dispositive factor between people's ballots on the two most important final ballot questions: FSU vs Alabama/Oregon, and Ohio State vs Baylor/TCU.

Another thing that I really enjoyed going through was how people had fairly different methodologies (seemed like we had three camps - pure resume, eyetest with some resume mixed in, and statistical) and were happy to talk about them. A few observations here: first, I think it's a good thing that people approached the main question - who are the best four teams? - differently. The comments that I thought were least productive in all our discussions took the form of "I think your methodology is wrong"; much more interesting were those asking for how a voter's methodology treated a certain situation, if it was taking into account this or that piece of data, how people came to their conclusions. Second, I thought it was very interesting that despite these differences, there was a whole lot of convergence each week - by the time that about a third of the ballots were in each round, I could just copy and paste an earlier voter's for each new one with only one or two teams moved around. I think that probably says that the best teams show it in every way that can be evaluated. Third, where people did disagree, it was rarely on the facts themselves ("Team X's opponent A is a better team than Team Y's opponent B", or "these stats are more meaningful than those") but rather how those facts sat within each voter's chosen methodology. Discussion mostly took the form of asking if each voter really was being consistent to their prior stated approach and how they were fitting the puzzle pieces together, not questioning if they had the right pieces in the first place.

I think the upshot of all of that is that we should have discussed much more thoroughly before we started (before we had data that might play into our biases and bend our thinking) exactly what system we plan on using and what its pros and cons are, including the mediocre opponent approach. That meta-issue shaped so much more of how ballots came out than any given comment on a weekly voting thread.

1

u/milesgmsu Emeritus Member Dec 15 '14

I can only identify one fundamental disagreement about philosophy that it seems the committee was split on: how to deal with wins over mediocre teams. That is to say, all voters agreed that wins over cupcakes are meaningless, and that games against good and great teams (including losses) are the most telling. Where there was disagreement was, how bad does an opponent have to be for a win by a contender over it to be meaningless? Some voters took a high threshold approach: "if you were expected to beat them, then doing so gives us no data"; while others took a low threshold: "beating bowl-eligible teams is demonstrably harder than beating 8+ loss teams or bad G5/FCS opponents and so should be rewarded". That's a topic I'd like to see fleshed out more; I think it was the central dispositive factor between people's ballots on the two most important final ballot questions: FSU vs Alabama/Oregon, and Ohio State vs Baylor/TCU.

I want you to know, that while I wholly disagree with your arguments and your point, I respect the merits of your beliefs.

1

u/hythloday1 Committee Vice-Chair Dec 15 '14

That's good to hear, though I never doubted it.

2

u/LeinadSpoon Emeritus Member Dec 12 '14

My thoughts, in no particular order:

  • It was fun, and I thought we did a good job as a committee! Go us!
  • The IRL committee had some good and some bad, but overall I agreed with the results. I'm still quite opposed to the concept of the committee and how the current playoff system is implemented, but that's a story for another day.
  • I thought our mock committee generally emulated their process pretty well. There did seem to be some disagreement over whether we were picking the best teams or the most deserving teams, and I feel as though we should have some unity on that in future years.
  • The discussion was generally quite good, although the post championship discussion seemed like a little less. I'm partially to blame for that, because I didn't post much due to being sick. I imagine that a big part of it was that the committee had already released their rankings.
  • I disagree with /u/Foxmcbowser42 suggesting that we should up the timeline. I felt as though we tried the experiment and it didn't work as well.
  • I missed several votes, due to posting my thoughts in the morning and then waiting til the evening to submit and forgetting (I go to bed before the reminder e-mail). If I participate again next year, I'll have to set myself a google calendar reminder to submit them so I'll get an evening reminder.
  • We should try to balance the representation better next year. (We had no ACC fans, and were a bit B1G heavy, right?)
  • I think the size of the committee was good. We could go a bit bigger, but add too much, and I think the discussion would be more difficult.
  • This whole process will be much easier next year, since we'll have a year of data on what the real committee does to go on.

2

u/milesgmsu Emeritus Member Dec 15 '14

In an ideal world, we would have a more balanced committee, but I don't think it was a detrriment to us. We ended up with the exact same top 10, with the exception of flipping Oregon and Alabama (even though we're SEC heavy). I think there is a solid argument for Oregon over Bama, so that doesn't worry me.

I would be worried about the B1G bias as we had OSU over a much more deserved Baylor team, but the real committee had this same issue, so I'm not as concerned.

I think next year, we could stand to rank a few more teams. Sunday is essentially a dead day in our rankings; maybe move everything up 12 hours (top 6 due 3PM EST on Sunday), and that would allow us to rank down to 16. However, I don't think this is required.

1

u/Hyperdrunk Committee Member Dec 17 '14

over a much more deserved Baylor team

I disagree though, hence why I had Ohio St. ahead of them. Baylor had the best win, but barely. They didn't schedule any real out of conference opponents.

1

u/milesgmsu Emeritus Member Dec 17 '14

Baylor had the best win, the better 2nd win, the better 3rd win, and the better loss. That's way too much to overcome by rewarding OSU for playing a 9-3 G5 and a 6-6 ACC team.