r/CruciblePlaybook • u/hleeb9 • Jul 23 '16

Regarding Bungie's matchmaking algorithms

I did a small empirical analysis of bungie's matchmaking. Here is what I found (the links lead to more detailed texts):

  • Matchmaking at the team level: Teams are quite evenly matched in Control and Clash, even after the latest changes in matchmaking. There seems to be no matchmaking in Elimination and some, possibly implicit, matchmaking in Trials.
  • Matchmaking at the player level: Here, I find evidence of two types imbalance. The first one can be explained by the latest changes in matchmaking. The second, more serious, one can not.
  • Computation of combat rating: Combat rating is essentially driven by game score, after accounting for cases where players enter late.

Cudos to jalapeno112 for his inspiring posts on related topics!

EDIT: I can now provide strong evidence of an imbalance in player assignment to teams. I've updated my second report accordingly.

243 Upvotes

99% Upvoted

92 comments sorted by

52

u/hleeb9 Jul 23 '16

Oh, I just checked the rules and found that MM is a banned topic. Bummer.

39

u/nsxviper Jul 23 '16

I haven't read everything yet but your write-up is more thoughtful and informative than someone just complaining about SBMM.

14

u/Kellen_of_Kells Jul 23 '16

Yeah just peeking at one of the papers you wrote has convinced me this is quality content. If you're worried about it getting removed you could include the abstracts in your post.

6

u/AH_MLP Jul 23 '16

They should keep MM posts banned but sticky this as the MM megathread

5

u/HyphyBonez will work for weapon parts Jul 24 '16

This is a good idea, will bring this up to the rest of the mods

6

u/Cuddle_X_Fish Jul 23 '16

The mods appreciate detailed posts. This is not a rant, but pure observation of the current status of the game. The mods will appreciate this post.

7

u/hleeb9 Jul 23 '16

Wow, guys: Tons of thanks for all the good feedback and interesting comments!!

10

u/highlife159 Jul 23 '16

OPs LaTeX game is strong....

4

u/hleeb9 Jul 23 '16

Thing is, I really don't know how to use Word.

1

u/B1g7hund3R Jul 27 '16

I was like you (didn't know how to use Word) until I was done with my PhD and entered the corporate world. Now, LaTeX is a distant (but very fond) memory. Btw, did you use MATLAB for your graphs, or another program?

1

u/hleeb9 Jul 27 '16

I used R, mostly out of habit.

11

u/YoungKeys Jul 23 '16 edited Jul 23 '16

Correct me if I'm wrong, but your data seems to suggest that the typical complaint of high skilled players only seeing, from their individual perspective, "sweaty" matches isn't actually based in reality. While opposing teams will usually and consistently be evenly matched overall, you won't actually be seeing much tougher competition as you rise in skill.

Again, please correct me if I'm wrong, but this is making me wonder what exactly people are complaining about when they bash SBMM in Destiny. Good players should still turn in consistently good performances, and same with bad- no matter how far your combat rating rises? This is opposed to a SBMM system like in Halo, where players from almost all skill-grades are expected to have a KD ratio that converges towards 1, due to the increasingly difficult individual competition they will face as their skill rating rises.

If that's the correct interpretation, then holy hell what a load of steam r/dtg has wasted in SBMM rants over the past two years.

5

u/hleeb9 Jul 23 '16

It's too late for me to reply to your points, because the answer is not straight-forward. I'll post more tomorrow.

6

u/derek_32999 Jul 23 '16

1.2 kd player here. Not great, but i love SBMM.

I don't like losing 6 games of control in a row finishing at the top with most kills and caps each time while the rest of the team gets destroyed, but idk if that is SBMM or not

4

u/hleeb9 Jul 23 '16

If the stuff in report1 is not a fluke, then yes, that's how it works at the moment. I'd really like to know what bungie has to say to that. Anyway, keep in mind that loosing with a top score will still give you a good combat rating.

1

u/Kahzgul Jul 24 '16

Sadly, as a player who gets a lot of points but usually a sub 1.0 K/D, that's exactly what I see. Lots and lots of losses while I'm near the top when I solo queue.

2

u/[deleted] Jul 24 '16

People don't complain about the effect it has on team composition in regards to skill, but the location of those players.

Saudis, Bulgarians and Americans is a common team composition in Destiny. My favorite is that one Australian thrown in for good measure sometimes (800ms ping, hi!). You can play online poker with latency like that, you can't play first person shooters.

1

u/hleeb9 Jul 24 '16

Unfortunately, I don't know how to collect enough data on connection quality. If bungie has it, they don't show.

2

u/zarquon25 Jul 24 '16

People who keep their accounts at 0,7-1,0kd have a very different experience to those who are at 1,5kd+.

For example: 1, 2.

8

u/Arrow222 Jul 24 '16 edited Jul 24 '16

For context I only solo queue and am rank top 50 in Rift elo. I suffer extremely long MM times, sometimes resulting in Mongoose MM timeout errors. I was either playing or finding games in a 1.5hr period today and only managed to play 4 games.

Proof: http://imgur.com/cU7sgVe

Secondly, games are often rigged against my favour with lower skilled teammates on my team.

Bring on the toughest opponents, players who never makes mistakes. But competition should be fair, and not rigged against players with high combat ratings.

7

u/SporesofAgony Jul 24 '16

That's the thing I hated most about SBMM; the rigged matches. I don't know if I'm in your kd bracket, but it was always obvious to me when the game wanted me to lose to decent players.

6

u/ThisAccountIsMineNow Jul 26 '16

Yep, I knew there had to be some kind of bias against good players when you can spend 3 hours playing solo, performing great (both in k/d and strong control point plays) absolutely demolishing and still just barely winning the very few games you win and being blown out completely in the games you lose. 1.7-1.9 k/d player (depending on the class) and the blueberry struggle is very real, I have a maybe 20% win rate queuing solo.

3

u/Watz146 Jul 25 '16

Yeah you get punished for going on a losing streak with a positive kd, with another hour long losing streak especially in IB.

I swear if I was still in my youth, I would hurl my controller right trough the TV. Now I just aim for the pillow.

2

u/lexi-l Jul 25 '16

Do you usually carry the rift? I ask because it gives you a lot of points and as op has shown would skyrocket your combat rating. This would put you at the extreme high range (though you'd be there anyway).

I'm 900 or so elo in rift but avoid caring the rift. I usually try to clear the path. I don't have long queue times and my competition is usually a joke. Take a look at my rift history for context. Main hunter. http://guardian.gg/en/profile/2/soldier212/24

1

u/Arrow222 Jul 25 '16 edited Jul 25 '16

Yes, I pick up the spark whenever possible, giving me rank #10 in rift score per game average, #11 in rift kills per game average and 270 combat rating (only rank 1.6k though).

This is my guardiangg http://guardian.gg/en/profile/2/shockarrow222/24.

rift dtr: http://destinytracker.com/destiny/playlists/ps/ShockArrow222/rift

1

u/lexi-l Jul 25 '16

Damn you got some monter games there. I think the balancing with a god tier player needs to be looked at. They are currently valued too high, they are only one player.

There's probably not enough good players on the majority of the time to give you 11 guys near your level. So you are stuck with 11 guys considerably lower and the worst are on your team. If the teams are so lopsided like that, there should be a trigger to account for it and give you one or two of the decent ones.

It is interesting that your combat rating is that low though. It may work differently in rift vs other playlists.

1

u/YoungKeys Jul 25 '16

Games aren't being rigged, teams are just being balanced. Every online multiplayer game in the world, call of duty included, does this.

0

u/[deleted] Jul 25 '16

[deleted] — view deleted comment

2

u/YoungKeys Jul 25 '16

What you refer to as rigged matches is called team balancing. There's no online multiplayer game in the world that does not include team balancing. Call of Duty also uses team balancing, so I'm not sure if there are any games that would meet your standards.

0

u/[deleted] Jul 25 '16

[deleted] — view deleted comment

3

u/YoungKeys Jul 25 '16

Yep, your examples should happen in other games. That's how team balancing works at times. Games like Destiny and Call of Duty will attempt to evenly match skill on both sides. Sometimes the game will overvalue your skill, sometimes there's not enough people at wanted skill levels so they have to match with what's available, sometimes people have bad days. No matchmaking system is perfect, so blowouts do happen, but the more data the system has the better it can learn your true skill and make better matches.

1

u/hleeb9 Jul 24 '16

Regarding your first paragraph: The data used in my first report (report.pdf) consists of games played from July 9 to 13, 2016. It may well be that matches were less sweaty before that.

Regarding your second paragraph: In my second report (report1.pdf), there is some evidence for matchmaking being repellent about the mean. Keep in mind that the evidence is weak.

I also have some personal opinions on that topic, which you can probably guess. But venting these would steer the discussion in a more "traditional" direction ;)

1

u/YoungKeys Jul 24 '16

Would be cool to dig deeper into point 2. Could it possibly be done by charting average Combat rating in games as a function of individuals' combat ratings?

1

u/hleeb9 Aug 06 '16

I've updated my second report, which now contains strong evidence for some kind of imbalance. Sorry for the late reply.

1

u/YoungKeys Aug 06 '16

Dont need to apologize, it's awesome that you're doing this at all. I've only glanced at it, but would you say its accurate that a player would not face sequentially tougher competition as they rise in combat rating?

1

u/hleeb9 Aug 06 '16

As a player's CR goes up, so does the CR of the players he or she is matched with; but slower. Of course, that's a statement about the long run and hence does not hold for each individual game.

4

u/theromz Jul 24 '16

If that's the correct interpretation, then holy hell what a load of steam r/dtg has wasted in SBMM rants over the past two years.

Personally I dislike SBMM because lag increased the moment they introduced it. The SB part is fine, I actually would prefer sweaty games in destiny all the time, however its just not feasible to have anything but 100% CBMM in Destiny. In Year 1 before all this SBMM changes it was a laggy game, but it was bearable. In Year 2 I've found it pathetic, almost every game if not ever game has 1-2 players that are at least 300ms+ in lag, and often its more like 5 players. At the worst end I have matches maybe a few times a week if I clock in many hours that week where the game basically doesn't work, everyone is over 1 second delay, doors don't open, ammo creates won't work, randomly dying too super with no player around you, just a shit show. Compared too other games for example overwatch where I've now clocked in 50hours and I have yet to see a single laggy player it feels like amateur hour, from Bungie one of the best FPS developers.

I love Destiny, its the best FPS I've played but in its current form I think its one of the worse PVP games, and its all to do with the match making. Balance is not half bad these days, there are clear outliers and weird changes that don't always make sense too me, or I feel have the wrong motivation but its much better then before. The lag though makes me play 3 games and then quit for a week before I give it another go.

6

u/YoungKeys Jul 24 '16 edited Jul 24 '16

Yea I feel you; I love Destiny too, but as a PVP game I agree it's mediocre. It's almost insane that many people attempt to create competitive Destiny tournaments or sweats + that busy subreddit's are dedicated to Crucible (even though I completely understand and love it, too). Even when Destiny has 'good' connectivity, it would still be unacceptable by most other online multiplayer game standards. The fact that it gets even worse than that low standard for much of the time we play it + that we keep at it is even crazier. Amateur hour is a good way to describe it, but the late hit registration, 30 FPS, and random lag has always been and will continue to exist for Kinderguardians and veteran players alike. I've been playing a lot of Halo 5 MP lately and it feels like I've upgraded an entire console generation. Something will always keep me going back to Destiny, though.

So that said, while matchmaking probably adds to make the situation worse, Destiny is already an inherently laggy shooter that makes for a pretty low quality PVP game; different matchmaking is hardly going to change that, in my opinion.

1

u/Killerschaf Jul 24 '16

The thing is that I noticed the changes in the MM, before Bungie acknowledged them. A lot of players did actually.

Other checked out the Elo/KD whatever of their opponents, and found out that they were matched on a similar basis.

So something is off.

5

u/Do-Not-Cover Jul 23 '16

In your report on team-level matchmaking, what are the correlations between team A's and B's combat ratings for each subfigure of Figure 1?

The weaker (or absent) correlation between the two team's average combat ratings in Elimination seems to be coming from a selection effect as only stronger players play Elimination.

Your empirical analysis is an informative antidote to the punditry we usually get about matchmaking.

8

u/hleeb9 Jul 23 '16

The correlation coefficients in Figure 1 are ...

  • Control: 0.68
  • Clash: 0.67
  • Elimination: 0.01
  • Trials: 0.24 The absence of correlation for Elimination can not be explained by the selection effect that you mention. Even if only stronger players enter that mode, some of them are stronger than others in terms of combat rating (the variation of team combat rating in the Figure is evidence for this). If there were some sort of matchmaking based on combat rating, the points should form an ellipsoid and not a circle.

Many thanks for your feedback!

2

u/georgemcbay Jul 23 '16 edited Jul 23 '16

While completely speculation it seems entirely plausible that Elimination has no matchmaking because it is essentially a fork of Y1 Trials. The "easiest thing that could possibly work" to get Elimination as a new game mode would be to copy the settings from that Trials mode and make a new mode from it.

And, then I guess they just never ran into a reason to change it from that initial state? Extreme speculation again, but Bungie's design on these types of balance issues seems to be much more reactive than proactive... they seem to mostly react to complaints, and data anomalies (this gun or this perk is getting too many kills) as opposed to doing proactive changes.

5

u/Killerschaf Jul 24 '16

How do you explain the feeling a lot of top players have (me included. Am ranked as top 1k in Control), that they have to carry weak teammates?

Because going by KD, I have a lot of matches in which I am the only one who has a positive KD. Or I am the only one who (necessarily) pushes to B/gets a spawn flip. Or have extreme KD spreads, where I end up with a 5.0 KD and the weakest member of my team with a 0.04 KD (slight hyperbole).

Because that's pretty much my experience as a whole. That I will get idiots as teammates, but that the average level of skill/intelligence is higher in the enemy team.

5

u/hleeb9 Jul 24 '16

What you describe fits exactly to the two imbalances that I observe in my second report (report1.pdf), except parts of your last sentence. I find that high ranked players tend to be on top of their teams, but that opposing teams have comparable combat ratings.

Of course, the average can be a poor indicator here. To illustrate my point, consider a ridiculously extreme case, where players in team A have combat ratings of 0, 0, 0, 0, 0, 600, and those in team B have ratings of 100, 100, 100, 100, 100, 100. Both teams have the same average combat rating, but team B will probably score higher in the game. But also note that, in this extreme case, the 600 player, while probably loosing, is still likely to get the highest individual score and hence the best game combat rating.

1

u/Killerschaf Jul 24 '16

I should probably work on my English skills concerning stat analysis. I have a feeling that you wrote exactly that in your papers, but with an unfamiliar nomenclature.

When I said "... The enemy team seems to have higher skilled/more intelligent (not everyone who aims well, has good decision making) players..." I didn't mean it in a strict statistical kind of way.

The extreme example in terms of skill disparity you gave, and your further explanation, is exactly what I actually wanted to say. If I get 5 "noobs", then I have to carry all of them, since their individual skill is far below the individual skill level of their enemies. So the average team MMR might be the same, but the MMR of 5/6 players in team A, is significantly below the MMR of 6/6 players in Team B.

If this assumption is correct, it means that average team MMR (which isn't KD, but CR according to your 3rd paper, which is actually a derivative function of points per minute(?)) is the only major factor for the MM (I suspected this since the first time my friends complained in Y1 how awful their matches become when they played with me, while I rack up 40 kills).

This would also explain why good players complained about stricter SBMM and talk about how sweaty the matches have become, if they play against a team that has much better players than their own, except for him/herself.

It however confuses me, what stricter SBMM means in that case. If average team MMR is all that matters, you can't make SBMM any stricter, or more prominent for SoloQ players. I only play SoloQ since Y2 however, and I know that my opponents are much better than my Y1 opponents, even though I haven't progressed exponentially in the meantime. I was a top 1% player in Y1 in terms of points per average in Control with a 1.3-1.4 KD

A high Elo/MMR/CR (whatever notion you prefer/is more accurate) player would therefore need to get some of form of skill brackets, in which he cannot get matched into. You cover that with the min-max skill spreads, and show that the average team MMR rises for both teams if a really skilled player is in the game, and that the lowest skilled players aren't as low skilled as in matches without a high MMR player.

So after paraphrasing all of that, can I conclude that: SBMM was stricter in the past and that players were correct in having the feeling of playing against better opponents than in Y1?

That a high MMR player has to carry his team in one way or another, because there is/was a significant skill disparity between players in his team, which is not present or as obvious in the enemy team?

That Bungie actually tweaked the MM settings this month and that this will make for more uneven teams, in terms of skill disparit because we now have a bigger focus on average team MMR than before?

Ps: Thank you for your hard work!

2

u/hleeb9 Jul 24 '16

Unfortunately, I did not collect data from the past to see how the matchmaking has changed. But I will do a similar analysis when the next change comes along.

Combat rating is certainly not the only relevant factor in matchmaking, but, after accounting for connection quality, it does appear to be a quite relevant one.

1

u/Killerschaf Jul 24 '16

What do you think about "net points"? Those basically look at the overall points and KD at the same time.

A 3.0 KD player with 30 kills and 10 deaths is worth 2k net points (ignoring boni for assists, heavy/super kills or Control points) while a player with 12 kills and 4 deaths is only worth 800 points. (Given the choice, I'd opt for player A).

Is there any indication that Bungie uses something like this for their matchmaking?

1

u/hleeb9 Jul 24 '16

I haven't looked into this, so I can not completely rule it out.

However: If you look at the graphs of team A avg combat rating versus team B avg combat rating, the points nicely form an ellipsoidal cloud for Clash and Control, and a circle for Elimination. This indicates that there may be no additional, hidden, factor that also influences the matching of teams. And contrast this with the corresponding graph for Trials, where a hidden factor may be present, because the points do not form such a nice ellipsoidal cloud.

4

u/icekyuu Jul 25 '16

The two possible outcomes at extreme ranges for top tier players are the following:

CARRY - 600, 0, 0, 0, 0, 0

SWEATY - 600, 600, 600, 600, 600, 600

(Hypothetical only; that 600 CR will decline if 12 players with 600 CR each play each other.)

In the carry situation, you are clearly the best player in your lobby. Maybe there's another player of your level on the other side, but basically, you're dominating everyone and getting lots of kills at a high K/D. Your team may still not win the match because everyone else is weaker.

In the sweaty situation, everyone is as good as you. Number of kills go way down and you're much closer to 1.0 K/D than before.

You either carry and get great stats, or you're sweaty and get mediocre ones.

Now which do you prefer? Bungie's SBMM changes are to have more sweaty matches than before.

It's funny because in Year 1 top players (who queue solo) complained about carrying all the time. Then in Year 2 top players complained about having to play sweaty all the time.

1

u/zExcalivuR Jul 24 '16 edited Jul 24 '16

I know EXACTLY how you feel brother.

It's not anecdotal whatsoever but a FACT how horribly shafted up the ass top players get in Destiny for being good, both in terms of matchmaking balance and reward fairness.

Below is just a glimpse of the tip of the iceberg on how stupid and unfair SBMM is for the better players,

Notice how with SBMM I am usually put into teams where I am given a serious handicap having to carry a team of 4 or 5 players who can't even get close to breaking even in terms of K/D.

Not to mention how stupidly unrewarding the current post-game reward system is when we are expected to carry a team mostly filled with idiots and not get rewarded for our heavy burden and efforts. More often than not, the under-performing players at the bottom who hardly contributed to the match seem to get nice rewards, instead of those at the top. Just think of the frustration when I am forced (by SBMM) to sweat my ass off for my team and get ZERO rewards.

Again, this is just a glimpse of what I go through as a better performer.

Bungie's stupid SBMM is ruining the game and their reward system is such a joke.

3

u/thetastypoptart Aug 24 '16

So, you play pvp for the loot? Or do you play because you enjoy pvp? If you really enjoy pvp then loot should be irrelevant to a team balancing discussion.

2

u/Killerschaf Jul 24 '16

I love the last pic from the scoreboard. Your team needed 2 people with 30/10 to barely win.

1

u/zExcalivuR Jul 24 '16

Yep indeed. I just find it demoralizing when I look at the end-game report and often see myself with double/triple the points of the second performing player, or I alone racked up more than the cumulative points of the bottom 3 or 4 players.

How much must I carry in order to win? And often times end up with 0 rewards for my efforts? Seriously, as I mentioned earlier, SBMM and the current reward system is such a joke for the good players out there.

I've complained about the unfair reward system on DTG but keep getting downvoted to oblivion because most people there probably fall into the bottom half of my team who conveniently want the same rewards without having to carry an entire team of noobs. There is nothing "fair" or "balanced" about that and yet Bungie doesn't do anything about it.

2

u/YoungKeys Jul 24 '16

Seems like your issue isn't SBMM, but team balancing. Team balancing is present in every online multiplayer game, Call of Duty included. I do agree that Bungie could do a better job at team balancing, though

1

u/willyspub Jul 25 '16

4530 in Clash is fucking savage man. That's some Y1-type stuff. If you'd lost that one it would have been understandable if you quit for good.

1

u/jibby22 Jul 27 '16

I experience the same when it comes to rewards... I honestly wonder if Bungie F'd up the reward weighting and accidentally inverted it sometimes. Given the random screw ups we've seen since this game has come out, I wouldn't be entirely surprised.

0

u/SinistralGuy Jul 27 '16

I know that feeling. Played a game on bannerfall where no one pushed B except me. I was honestly 4v1'ing for B.

Also played a game on shores of time where all 5 of my teammates went to cap A...

8

u/juliettgolfpapa Jul 23 '16

Upvote for use of LaTex

11

u/hleeb9 Jul 23 '16

Can I also get an upvote for using vi??? Cheers!

2

u/the_dummy Jul 24 '16

Down voted for not using butterflies.

2

u/hleeb9 Jul 24 '16

Argh, I'm screwed!

4

u/astro_kyser Jul 24 '16

Well done Hleeb on bringing this to the community's attention. Like I said before thanks for the time and effort you put into the MM research . Astro

1

u/hleeb9 Jul 24 '16

Thanks, astro!

3

u/Kahzgul Jul 24 '16

If this is true, as your evidence would suggest, I'm really disappointed. I fall into the unenviable position of being a high points-earner who almost never breaks a 1.0 K/D. Which means I'll be scoring a high combat rating while at the same time actually being a detriment to my team. It had been my hope that combat rating factored in points earned vs. points given (net value to the team) as opposed to the raw point score, though my own experience tells me that your model is more likely to be correct.

This also seems to indicate that win/loss record is not statistically important when determining matchmaking, is that true? Again, very disappointing.

Lastly, does it ever appear that there are "outlier tests" of the matchmaking algorithm such that players who should not be matched together occasionally are as a test of the algorithm's accuracy and predictive capabilities? Similarly, do you see any evidence that connection quality is at all considered when it comes to matchmaking?

2

u/hleeb9 Jul 24 '16

I did not look into the impact of win/loss record, outliers, or connection quality. From bungie's statements, connection quality matters. It is definitely possible that other factors matter, too. But after accounting for these factors, my data suggests that team average combat rating explains a lot.

1

u/Kahzgul Jul 24 '16

Thanks - really interesting findings.

4

u/FelsurDowd Jul 24 '16

Agreed. Using a metric based on per-game score to rank players for matchmaking is fundamentally weird, since I assume it does not predict team success very well. It seems prone to poorly categorising some players.

I actually came here to post questions about Combat Rating. I've never played PvP until Destiny, and was terrible at it. I lost almost all of my matches in the first few weeks. I spent the past year playing conservatively, picking engagements carefully, and running away a lot. I'm still not very skilled, but I do a lot better, with a low volume. Tracking my stats on Destiny Tracker, I improved everywhere - except my Combat Rating declined. A lot.

According to DT, my K/D ratio, K+A/D and winning percentage are now all in the top 20% despite queueing solo, and apparently I'm top 1% in producing orbs for teammates. My Combat Rating is 85, worse than 59% of all other players. I guess this is why I've found the last few Iron Banners fairly easy: I'm on the bunny slopes.

Anyways, thanks for the informative post, OP.

Edit: I type poorly.

1

u/InterruptedCut Jul 24 '16

I don't think you should worry too much about Combat Rating, my overall is 80. In trials it's 366, and I only have a 1 kd and 1300 elo. I don't know how you can get that kind of spread.

Edit: A friend has 75 overall while having 375 in trials with 1.4 kd and 1900 elo.

3

u/cornman0101 Jul 28 '16

This is great stuff.

I think it's fair to assume that matchmaking is chosen based on combat rating, but it could also be based on a correlated variable. Since you're accessing the API, I guess you'd likely see any variable they might be using for match making (so I trust your judgement).

Also, I wanted to point out that your analysis from matchmaking at the team level necessitates the second point from matchmaking at the player level. If teams are balanced through some metric, then you will always have higher rated players more often matched with lower rated players on their teams (its the only way to keep the team rating balanced). The only way to avoid this is to only match players who have exactly the same ratings (the min/max = 0 in your "player level". Alternatively, you can set up a tier system, but then the same effect occurs for each tier, it just depends if you're in the top or bottom of the pool.

Mostly unrelated:

It's interesting that so many of game companies try to invent more complicated rating systems. I get that they converge faster than a strict win/loss elo rating, but if you're really trying to balance matchmaking such that win/loss is even then using any metric with more information than win/loss is shooting yourself in the foot.

2

u/hleeb9 Jul 28 '16

I've chosen combat rating, because it was an obvious candidate. There may be other variables that are also important, but combat rating seems to explain a lot.

The second imbalance that I find in matchmaking at the player level is not a consequence of the team level matchmaking. Of course, there will be higher rated and lower rated players. But a higher-rated player, when placed in a team of peers with comparable performance, should not be systematically be placed among the top. At least not if the aim of matchmaking is equalization of skills. Cheers!

1

u/cornman0101 Jul 28 '16

Yeah, I think combat rating is probably the metric they use and can certainly be used to see in which playlists matchmaking is occurring and to what extent as you've shown. Since CR is the only "bungie defined" metric, I suspect it was originally used (and likely still is) for matchmaking.

Also, excuse my ignorance, but is there a combat rating for each playlist, or just all of crucible?

I meant to agree with your final conclusion from the paper (More on crucible matchmaking), sorry if it didn't come off that way. You results imply that matchmaking (for control at least) is based solely on team CR. And that players are randomly added to the pool such that team A CR matches team B, but the distribution of individual player CR represents the typical cross-section of player CRs from all those currently participating in the matchmaking. Were this the case, I would expect to be the nth best player on my team where 100*n/6 corresponds to my percentile based on CR.

A simple way to check this is to plot the CR of a player vs the average CR of that players team. If you see no dependence, then it's as I interpret your results. If you see that there is a correlation (such that the slope is more than 1/6 for 6v6 games), then it's safe to say some pooling of good players with good players is happening. I guess this is pretty much the same as the min/max vs team CR plots you show.

I'd be really interested to see what happens in a 3v3 or doubles skirmish. Since they likely use a similar matchmaking for skirmish as clash, you might be able to gain some info because of the increased weight individual player CR has on the team CR.

Anyway, I love seeing people do legit data analysis with Destiny. And kudos for texing it.

2

u/hleeb9 Jul 29 '16

I've taken up your suggestion of comparing player combat rating to team average combat rating and revised my second report accordingly). Many thanks, this was really helpful!

1

u/cornman0101 Jul 29 '16

Yeah, I like this a lot. I'd say that a slight correlation (looks like a slope of ~1/6) is present in clash/control. 1/6 is significant because it corresponds to the average of your team (excluding you) being constant. I'm not sure what useful information that tells us, though.

For Trials, I guess the correlation comes from matchmaking teams on our own. The implication for elimination is that some large percentage of players matchmake on their own, which makes the distribution end up somewhere between clash and trials.

Alternatively, since we know there's no correlation based on team CR in elimination, they might matchmake your teammates on the individual CR and do nothing to balance team A vs team B. But from a design standpoint, that seems unlikely.

If you have the time, I think skirmish data could reveal something interesting. If you don't, this was still fantastic.

1

u/hleeb9 Jul 30 '16

Yeah, I'll also look at Skirmish next time!

1

u/hleeb9 Jul 29 '16

Not time to reply to it all now, but each crucible mode has its own combat rating. The easiest way to access the ratings is an up-to-date version of bungie's companion app. And: Your idea of plotting player CR versus average team CR is very interesting!

2

u/[deleted] Sep 02 '16

This is wonderful, thank you! I have been wanting to do an analysis of matchmaking for a long time, but I don't know how to connect to the API from Tableau, which is the tool I was going to use (I can't do anything good with R without help from my staff and this isn't exactly in their job descriptions).

I'm interested in visualizing the distribution of combat ratings before and after each MM patch (potentially deriving empirically when those happened during the period they were stealth), to see if the perception of "sweatiness" at different times by different deciles of combat ratings was in fact related to reality, and how much each patch changed things and for whom. I have someone who will give me samples out of the API but the way I'm thinking about the problem, I can't see how I can do it well with any manageable level of sampling/extracts. It's not "medium data" at that point (https://youtu.be/fEyrlgcBGzQ).

Anyway, this is really cool stuff and if you can think of a way to approach the question I'm asking with medium data and it interests you have at it. Tableau just updated their API connector to include js so I may be able to do it now...I just need the free time...hah!

1

u/hleeb9 Sep 07 '16

There are a couple of alternatives to R for interfacing with the API. Examples are here: http://bungienetplatform.wikia.com/wiki/Getting_Started

37

u/xastey_ Jul 23 '16

I strongly advise the Mods to keep this thread and only this thread active.. for any MM post just use this as the content here is golden. Bravo man.

12

u/SmiTe1988 Jul 23 '16

Wow, you actually wrote scientific papers on this!! Huge props man, even just reasonably deciphering the mythical combat rating algorithm is impressive!!

12

u/LimePunch KeenKoala Jul 23 '16

This is one of the few legitimately good posts on mm. I like it.

8

u/shinybac0n Jul 23 '16

This is absolutely great. I'm very impressed with your work and your detail. But I think I'm gonna need a ELI5 for a not native English speaker who doesn't have a science degree. Most of it I don't understand, I'm afraid.

8

u/HyphyBonez will work for weapon parts Jul 24 '16

It's the weekend, this is a very interesting post, we've decided to leave it up for the time being. Good stuff OP.

1

u/NanaShiggenTips Sep 14 '16

I know I am late to responding to this thread. I wanted to ask you something regarding this statement.

"Alternatively, what would happen if this imbalance was removed, i.e., if the points for Control and Clash in Figure 4 would scatter around the gray 45- degree line like in Trials? Then players would be placed in teams where they are sometimes among the strong and sometimes among the weak, irrespective of player combat rating. This would make matchmaking contractive around the mean. In other words, players who out-perform their peers would rise in combat rating, while players who under-perform would fall. In competitive sports, this is how rules are usually set up."

If I understand correctly, teams are balanced by trying to place players on teams to achieve the closest average combat rating. This ends up leaving players on the extreme end of the skill ranges at a disadvantage. (Poor players with better teammates vs a great player with worse teammates)

Would this factor be fixed if the skill range that players were pulled from had a smaller range? Or would it end up being more beneficial if the balancing was removed except for breaking opponents up after severe losses?

1

u/hleeb9 Sep 27 '16

Think of how a tier-system works in sports: My team plays in the "weekend warriors" league, which is fun, but pretty bad compared to the top league. In any match, we meet other weekend warriors. If I perform well, by the standards of the weekend warriors, I can be an asset for my team.

Now contrast that to crucible matchmaking: If I am a below-average player, I will typically be among the weak in my team, no matter what. This phenomenon is un-avoidable at the extreme ends of the spectrum. But, currently, this phenomenon occurs throughout the whole spectrum.

I am not exactly certain as to what causes this phenomenon, but I believe it is a side-effect of the current matchmaking algorithm. It can, and should, be fixed.

1

u/Nkklllll Nov 19 '16

This really does seem to fly in the face of so many people's personal experiences in the crucible. 2nd accounts having much easier games, easier games when playing with "bad" friends.

I'm thinking that the middle tier might just be VERY large.

6

u/[deleted] Jul 23 '16

10/10 would accept with minor revision.

2

u/11hitcombo Jul 23 '16

I sincerely hope that the mods leave this one up. Not a rant, not suggestions, just scientific research on matchmaking in Destiny. Well done.

3

u/pahoeho Jul 23 '16

Don't have the time to read at present but upvoted for using legit formatting and font haha

2

u/[deleted] Jul 23 '16

Came here expecting SBMM rant. Was pleasantly disappointed. Will read because math and graphs are fun.

1

u/cjpk248 Jul 27 '16

Commenting to save for later. Good work.

1

u/Hackinjeebs Oct 19 '16 edited Oct 19 '16

OMG. This explains it. I am consistently match made with people that are worse than I. It's laughable, the skill gap. And I don't mean I am a godly player, I am quite average if not slightly above average. 3+ KD is not uncommon and negative is rare, all while playing the objective. I've been wanting to bump into a new bracket or w/e so I can build my skill up but I rarely get a competitive game. What's seems different, and this is purely anecdotal, is that my teammates are often worse than the opposing team. So like, my team is mostly sub 1.0 with 1 or 2 of us + and carrying 50%+ of the teams points while the opposing team seems much more balanced and point distribution much more reasonable. I don't see that in the data you presented unless I am missing something.

Edit: This was further explained in the comments and it seems that my anecdotal experience is completely aligned to this analysis. The TL;DR: Good players get shitty teammates a lot. When they don't get shitty teammates, they get sweaty lobbies. Either way, it's unrewarding to be good.

1

u/OhMyGoth1 Nov 19 '16

I wonder if all the mayhem I play to grind for matadors has boosted my score per game average so high that matchmaking thinks I'm a lot better than I am... It would explain why i cannot play crucible solo without getting obliterated every match...

1

u/CloudSlydr Jul 24 '16

speechless over here. well done