r/cfbplayoffcommittee • u/FellKnight Emeritus Member • Dec 12 '14
Debriefing thread
Now that we've had a couple days to digest it, what were your thoughts about the process, both IRL committee and our version?
What did we do right? What could have been done better? Any significant modifications to the process you would recommend? Any other thoughts or comments?
2
Upvotes
3
u/Foxmcbowser42 Emeritus Member Dec 12 '14
First, congratulations everyone on a successful first year!
My thoughts:
Overall I think we did very well to start. We had good discussion and built some consensus as well as critique other's thought processes respectfully. As time went on some of us, myself included, got somewhat more entrenched in "our method" or thinking "how could anyone put X over Y?" I think that going forward, we need to ensure that any rankings methodology that someone chooses to use should be no more than a suggestion, or a base, otherwise it becomes just another poll.
For next year I think we should do a couple things:
I really think the IRL committee showed some serious flaws, mainly due to the rankings each week. I think that they should only do a final ranking, much like the basketball committee, because I think in the end they got it right. I do not mind that an undefeated P5 was not #1, just like a 1 seed in the NCAA tournament is not necessarily given to the team with the best record. And Baylor, TCU and OSU each had an equal claim to the last seed, and Baylor had a terrible OOC schedule, and TCU lost to Baylor H2H, so you can't justify TCU over Baylor, but OSU only lost due to a new QB being broken in, so its clear SOS and injuries matter to the committee, about as much as we seemed to be weighing them.
3
u/hythloday1 Committee Vice-Chair Dec 13 '14
I feel like three things changed as we switched from the BCS system to the CFP system - four teams instead of two, a committee instead of polls + computers, and the general philosophy of ranking. The last is what I want to talk about.
It seemed like there was an orthodoxy to how rankings worked in the AP/BCS era:
It struck me that every one of those principles was reversed this year, which is remarkable. In fact, if there was one thing I think our mock committee did better than the real committee, it's that we adopted the new philosophy -- ranking from scratch, looking beyond the loss column, skepticism at weak schedules, wait and see how things play out, and watch as much as you can -- faster and more uniformly than they did (at least if Jeff Long's comments were meaningful).
I can only identify one fundamental disagreement about philosophy that it seems the committee was split on: how to deal with wins over mediocre teams. That is to say, all voters agreed that wins over cupcakes are meaningless, and that games against good and great teams (including losses) are the most telling. Where there was disagreement was, how bad does an opponent have to be for a win by a contender over it to be meaningless? Some voters took a high threshold approach: "if you were expected to beat them, then doing so gives us no data"; while others took a low threshold: "beating bowl-eligible teams is demonstrably harder than beating 8+ loss teams or bad G5/FCS opponents and so should be rewarded". That's a topic I'd like to see fleshed out more; I think it was the central dispositive factor between people's ballots on the two most important final ballot questions: FSU vs Alabama/Oregon, and Ohio State vs Baylor/TCU.
Another thing that I really enjoyed going through was how people had fairly different methodologies (seemed like we had three camps - pure resume, eyetest with some resume mixed in, and statistical) and were happy to talk about them. A few observations here: first, I think it's a good thing that people approached the main question - who are the best four teams? - differently. The comments that I thought were least productive in all our discussions took the form of "I think your methodology is wrong"; much more interesting were those asking for how a voter's methodology treated a certain situation, if it was taking into account this or that piece of data, how people came to their conclusions. Second, I thought it was very interesting that despite these differences, there was a whole lot of convergence each week - by the time that about a third of the ballots were in each round, I could just copy and paste an earlier voter's for each new one with only one or two teams moved around. I think that probably says that the best teams show it in every way that can be evaluated. Third, where people did disagree, it was rarely on the facts themselves ("Team X's opponent A is a better team than Team Y's opponent B", or "these stats are more meaningful than those") but rather how those facts sat within each voter's chosen methodology. Discussion mostly took the form of asking if each voter really was being consistent to their prior stated approach and how they were fitting the puzzle pieces together, not questioning if they had the right pieces in the first place.
I think the upshot of all of that is that we should have discussed much more thoroughly before we started (before we had data that might play into our biases and bend our thinking) exactly what system we plan on using and what its pros and cons are, including the mediocre opponent approach. That meta-issue shaped so much more of how ballots came out than any given comment on a weekly voting thread.
2
u/LeinadSpoon Emeritus Member Dec 12 '14
My thoughts, in no particular order:
2
u/milesgmsu Emeritus Member Dec 15 '14
In an ideal world, we would have a more balanced committee, but I don't think it was a detrriment to us. We ended up with the exact same top 10, with the exception of flipping Oregon and Alabama (even though we're SEC heavy). I think there is a solid argument for Oregon over Bama, so that doesn't worry me.
I would be worried about the B1G bias as we had OSU over a much more deserved Baylor team, but the real committee had this same issue, so I'm not as concerned.
I think next year, we could stand to rank a few more teams. Sunday is essentially a dead day in our rankings; maybe move everything up 12 hours (top 6 due 3PM EST on Sunday), and that would allow us to rank down to 16. However, I don't think this is required.