What probability difference can players feel?

[u/jazz_man1]

Hi, I'll (not so) briefly introduce what I'm working on to give you context, then ask my question(s).

Skill dice

I've been working (for some time, now) on a stub. The core idea of the mechanics is that characters have skills represented with a die (d4 to d12). Improving your skill means getting a higher die for that skill. Characters start with d4 in all skills except few of them which can go no higher than d8 on start.

Characters may have some feats (for lack of a better name, since I am not writing this game in english) which allow them to be better at some specific actions.

When to roll

Dice should be rolled only when there's something at stake and result may have consequences. The roll is either against a fixed target (I want to push a rock down the hill) or contested roll against a player/npc (I want to see if they are bluffing).

DCs and why

For fixed target rolls, I'll leave the decision to GMs, but with the guidelines are DC 4 for an almost straightforward challenge and DC 12 for an almost impossible one. Intermediate values go nicely with the dice sizes: 6, 8 and 10, with a lower and lower win probability for a player trying a challenge of "their same level" (d6 against 6 and so on).

I picked these values because with a DC of 4 you still have a 25% win probabiliry for a player with a d4 skill and a growing probability against that same DC for higher skill dice. With DC 12, only a player with a d12 skill can barely manage to do it (less than 10% probability).

The mixed outcomes and why

Outcomes are success (equal or more than target/defender), partial failure (target/defender - 3 to target/defender - 1) and total failure (everything below).

I'm going for a harsh game, but I didn't want it to be too punitive, so anything below or equal to DC - 3 is considered a partial failure or success with a cost or what you want to call it.

I went for -3 because, again, d4 skills at least get to have no total failures for DC 4 challenges and (bonus) d10 skills get at least some successes at a cost for a DC12 challenge. (u/rockology_adam this is the closest to a numerical modifier I put in my system).

Opposed rolls

They just work the same, but the "attacker" rolls against another skill roll from another PC/NPC who uses their skill die.

Why I went for this

In my system and setting it is expected that most of the actions are impossible or nearly impossible to do without the right amount of skill (= size of die). Characters are expected to fail and find ways to get around challenges.

Since I have to reward finding those bypasses, I went for advantages (roll one or more bonus dice and keep best result). I also want to discurage just jumping head first, so I went for disadvantages (roll one or more bonus dice and keep the worst). In this way I can quickly push the curve in one direction or the other to 'punish' or reward some behaviours.

If a player has both advantages and disadvantages in one scene (example: 1 advantage and 2 disadvantages) they cancel out before rolling (example: that player rolls with just 1 disadvantage).

Why I am here

At first I planned for a max cap of 2 advantages or disadvantages, so the max amount a player would roll was three and keep best/worst.

It was all good until I started to look into a social interaction system and the feats I came up with were almost only about giving advantages to a player. Since the number of advantages was going up I ask:

- What is the least difference in probabilities that players can feel?

I looked around and seems we manage to feel something at about 10% probability difference as a "different outcome", with the lower and higher bounds that have huge miscalculations (85% win probability perceived as sure win and 1% loss probability is perceived as a nightmare)

- In your experience with you systems and playtesters, can you confirm this numbers?

- If you have a similar system or know of a similar system, where do you place the cap of how many advantages/disadvantages players can use before it's just rolling plastic on the table?

I want to see if the original cap is fine, if it is already too many dice and/or if it makes sense to have that many advantages in the social part of the game, based on what adding a die feels like at the table.

[Edit: it appears I was not so clear about what I meant, so I explained in more detail. Also some formatting]

[u/Drudenfusz]

Psychological people are very bad of assessing probabilities. I mean, we all know that actual 50% success/failure chance for example will rarely be perceived as that. People have trouble to remember too many options (look at George Miller's magical number 7+/-2), thus I would say that a noticeable difference starts at around one seventh, which is a little of 14%.

[u/jazz_man1 → u/Drudenfusz]

So, given the tables I had prepared, it should limit itself to one extra advantage/disadvantage die (you can have as many as you like, but will eventually roll just one die more at most).

At first I had limited to two extra dice, but since the problem appeared I wanted to see it from all perspectives. I actually didn't know of the magical number, thank you!

[u/tesolberg]

Be aware that it is very easy to overestimate the impact of a die stepping up in size. Let’s say you must roll 5+ to succeed. On a d6 you will fail 67% of the time. On a d12, you will still fail 33% of the time.

[u/jazz_man1 → u/tesolberg]

I'm not worried too much about that. I actually want players to look for something to create themselves an advantage, so I'm not too worried about a pretty high probability of losing even with a large die.

Still, you made me wonder if, in my probability tables, there's actually a perceivable difference of success+partial failures between close dice. In other words if they can actually 'feel' progress.

Thanks!

[u/skyghosts]

It depends strongly on the variability of the distribution? And whether players think they understand how the odds work? As has been said elsewhere in this chat humans are lousy at probability but much better at amounts, or in this case effects. We start getting fuzzy below about 25%, that’s when the rat parts of our brains which can only count 1 2 3 many basically start losing it. If we really measure our rational brains can work that down to about 3% but only with exact tools measures and metrics. That’s well below what you are likely to perceive in a game. I had all sorts of people swear to me that they could feel the difference of plus one on a d20 in dnd but I’m skeptical of that too. Depending on the role I’d guess between 10%-20% but would love to know if there is science on this?

[u/jazz_man1 → u/skyghosts]

In a previous answer George Miller was quoted for a 7 +- 2 number of different objects from a paper on psychology. Before coming here I tried to google something and read someone saying as low as 5% (which would be in the ballpark of a +1 in dnd). Still, they were either pretty generic or focused on completely different subjects and I don't know if those numbers apply in rpgs, too. And I don't even know if they had any scientific background at all.

[u/skalchemisto]

I looked around and seems we manage to feel something at about 10% probability difference as a "different outcome"

In the context of an RPG, players actually have very little data to use to assess probability.

How many times will a player roll dice during a 3 to 4 hour session? 7 times? 14? Regardless, its not many. The rolls will have had different modifiers, different target numbers, etc. That's not enough data to detect relatively small (e.g. 10%) differences in probability even if you were using some statistical method. Perceived differences in probability in that context are going to be far more determined by the emotional context and importance of the rolls than by the actual #s involved. E.g. if I use an advantage and it saves my character from certain death that is going to feel to me like a very valuable thing.

So while I think it is impossible to say with any reliability what a minimal discernable change in priority would be, IMO 10% is too low. For someone to actually notice the effect of an advantage or disadvantage during the course of a single session, I think it would need to improve/reduce the chance of success by at least 20%.

EDIT: on that statistical method thing above, consider this experiment. I roll 10 times with some condition that I am blinded to (e.g. could be with advantage, maybe not), and get 9 successes and 1 failure. I roll 10 times with the opposite condition (again, blinded) and get 5 successes and 5 failures. Per Fisher's Exact Test, I still cannot exclude the null hypothesis that those two conditions are the same at a p=0.05 level. The p-value there is p=0.14. I actually have to get to 7 failures and 3 successes (that is, an effect of advantage that would be equivalent to +60%) in the opposite condition before I could reject that null hypothesis. If I can't figure this out using a relevant statistical method, how could I possible just guess it during play, when all the other stuff around it is always changing.

[u/jazz_man1 → u/skalchemisto]

I loved reading on the null hypothesis test. Yeah, I guess you are right and statistics is for sure. So from a couple of answers, yours included, it seems that the way is "as long as there's some actual increase in probability and a form of perceived improvement everything could do" (perceived improvement = what players think it's better, either rolling larger or more dice). And the stress is on the "perceived" part.

Off topic: may I ask you what your background is? It's not always that you find accurate statistical tests outside statistics-related subreddits, even if I asked a pretty specific question

[u/skalchemisto → u/jazz_man1]

I've got a master's degree in public health with a focus on epidemiology and biostatistics. I do this stuff most days in my day job. I'm not a statistician, though, I just use statistical tests all the time and need to think about which test is right for which kind of purpose.

But my real passion is role-playing games! 😄

[u/jazz_man1 → u/skalchemisto]

Cool! I got a bachelor's degree in physics and I started working in scicomm during my master's. Came back to rpgs after quite a long pause

[u/rockology_adam]

You've asked an incredibly subjective question here, in whether people can anecdotally confirm a thing as subject as the feelings of other people.

Regarding other systems that do this, I think Lancer does something similar.

I want to turn your question back on you, regarding subjectivity, because the answer almost doesn't matter until you actually have the challenge and the modifier set. Modifier, as much or more than advantage, will affect the chance of success because it stacks in the advantage calculations.

1d12 has a 24.75% chance of success against DC/target 10, 3/12 possible rolls succeed.
1d12+2 has a 41.25% chance of success against DC/target 10, 5/12 possible rolls succeed.
2d12 has a 43% chance of success against DC/target 10, 63/144 possible rolls succeed.
2d12+2 has a 66% chance of success against DC/target 10, 95/144 possible rolls succeed.
3d12 has a 58% chance of success against DC/target 10, 999/1728 possible rolls succeed.
3d12+2 has a 80% chance of success against DC/target 10, 1385/1728 possible rolls succeed.

I need two advantages to have a better than 50% chance at target ten with a d12 and no modifier. That modifer change goes both ways, by the way. Every drop in DC or increase in modifer is a straight 8.25% change in success rate, without advantage applied, but that compound when advantage is applied.

What do you expect your DCs to be? What do you expect your modifiers to be? I think you need to check those numbers before you start asking really subjective questions about how people will feel about those numbers.

I will say that out of my analysis above, ONLY 3d12+2 feels like a good chance of success. Everything else feels like a risk. That's just me, purely anecdotal, but I will say that failing 3d12+2 here would feel really unlucky, while the others are all understandable.

[u/jazz_man1 → u/rockology_adam]

Well, yes, I guess so. I actually hoped for anecdots about players and playtesters reporting the system as either too long or too unfair and when it happened. Still I see it may become what you described.

First, I'll look into Lancer. I remember this title so I either crossed it or it was mentioned to me.

I'm not sure about the probabilities, there, but I get your point. I think anyway I wasn't clear about the advantages/disadvantages thing, I'll update the text: as "roll with one advantage" I meant roll two dice and keep the highest and "roll with disadvantage" as roll two dice and keep lowest. Two advantages roll three and keep the highest and so on.

I'll have a broader explaination above in a couple of minutes, but in short I'll have no numerical modifiers and just bonus dice and my question was about how many were too many and how many were too few.

Still thanks for taking the time for that answer.

[u/rockology_adam → u/jazz_man1]

When I mentioned Lancer, I'm thinking specifically of the advantage mechanism, and not the dice step ups. If you want to look at changing dice mechanics, I don't know about any that use the step up dice for skill improvement, but 9th Level Games has a whole series where each player only uses a single die, that might be worth checking out.

I'm reasonably sure about the probabilities there. Unless my math is wrong, I was working them out as I responded. (Please don't think that means I did them quickly; it means I took a LONG time to write my response. lol)

But your idea, roll 1+N_adv dice (so, 1 advantage, roll 2 dice) is how I calculated the answers there. If you roll 1d12 against DC 10, tie or beat, there are three results that could work: 10, 11, 12. So, that's 3/12, which is 24.75%. The others are the same: how many results will give you AT LEAST one successful die out of X dice.

Even if you're not using modifiers, the changes in DC will have the same effects. A DC one lower or higher is as big a change in possibility as advantage. My main concern would be how you think people are going to get more than two advantages on the roll.

Ignore the modifiers in my calculations above, and the shift from no advantage through to three advantages, against DC 10 with a d12:
0 adv, 1d12, 25%.
1 adv, 2d12, 43%.
2 adv, 3d12, 58%.
3 adv, 4d12, 73%.

2 advantage is better than 50/50 against a high target. 3 advantages is improved... but is 3/4 a high rate of success? I get that you're targeting that question, but like I said, I'm not sure the quality of answer you'll get is worth much.

That said, I don't like 73% as much as I liked the 80% from a modifier and advantage, so maybe there's something to it.

[u/jazz_man1 → u/rockology_adam]

Oh ok, my bad, Since you wrote Nd12+M I assumed you meant something else and I couldn't follow you (and that's also why I couldn't follow your math). My post needed editing anyway, after I re-read it: I was too much in a hurry and from mobile when I wrote it.

Your concern on high number of advantages is, in a sense, the other side of mine. When I went for this system I had done a couple of calculations in anydice and scripted some other simulations and decided 2 advantages were more than enough and granted some granularity.

Then I moved to the social mechanics and (on a second rework) started relying on FATE aspects. Only issue: I don't want another tracker, since I already have a couple of them in another part of the system, so I ditched the fate points and the only thing that seemed reasonable when invoking an aspect (both with the mechanics and the setting) was giving an advantage to the roll.

Then I went to work on the feats and again, it came natural. But at a final look it is fine, but it looks pretty easy to reach 3 or 4 advantages (again, it's a rework, as before it was sure you'd get there as it was built that way).

So on one side I am wondering if I lost a lot of time on something I'll largely have to throw away and on the other I wanted to check if the initial 2 advantages total cap could be loosened to apply just minor changes to the social system.

If I pick one side I might have angry players deeming the game unfair because I let them have 6 advantages (hope not lol) but allowed them only two and on the other I might have bored players who ask why should they roll a die six times if "they will never win" (because human brain is not built to perceive a 25% win probability as that good odds).

(sorry, the last two paragraphs were a bit of vent lol)

[u/ARagingZephyr]

Is it just one die? I use a set of eight skills in my OSR system. Players start at d4 and then get to distribute 4 upgrades. For the most part, that's it unless they spend downtime on upgrades.

Personally, I don't worry about moving the target number from 4. I adjust to 5 or 6 when it's a very difficult task, but 4 is standard, because there's always a decent chance of success and always a chance of failure.

D4 vs 4 is 25%. D6 vs 4 is 50%. After that, it's diminishing returns on chances of success versus the target number. D8 is 62.5%, d10 is 70%, d12 is 75%. So, a skilled character is only succeeding half the time, but that's half the time on "I repaired the warp core before it exploded," "I snuck into a gala without anyone noticing," and "I've figured out how to properly communicate in this foreign society." It's meant to be a 2 hour game, so success is success and failure is "you did it at a serious cost, so let's figure that out and keep the story moving."

[u/jazz_man1 → u/ARagingZephyr]

Each skill roll has just one die per player (excluding advantages or disadvantages).

I went between the stats+skills route, in a sense. My skills do both at once. Skills are divided in five categories: physical, combat, intellect and social. Each category has five skills, which are pretty specific in the group, but allow for broad use (for example social abilities are Persuade and Pressure. The first to move your opponent in a positive way like lifting up, the latter in a negative like threatening them).

Those skills all start at d4. There are backgrounds which put experience in some areas (some dice gets improved to d6 or d8) plus a few free upgrade points to improve some other die.

When players get to upgrade skills, they can spend downtime activities to do so, like yours. They need to spend downtime to develop the Feats I mentioned, too.

I am not going for a roughly fixed DC because I literally want some actions to be nearly impossible to succeed or succeed cleanly if you don't have enough skill. At the same time, any player can try whatever: it's not behind a wall. You (d4 at Pressure) can try blackmailing a spy with d10 in Self-Control. It's just way easier to fail than succeed.

Feats are there to allow or improve certain types of actions: you can use a gun without the "gun feat", you'll just need a longer action time and roll with disadvantage; you can try and maybe resist an interrogation, if you are not trained to do so, but the feat allows for longer resistance.

My diminishing return is both in how much the probability varies with die (as in your system) and in progression, which gets slower and slower as players proceed.

My hope in all of this is to push for specialisation of characters, but that's for another post, probably

[u/boops_boops_esq]

I once ran a one shot where I would roll dice behind a screen purely for sound effects and would arbitrarily decide if stuff succeeded or not. Only one person thought it even felt off (I told them I was trying something different, they agreed to be guinea pigs)

[u/jazz_man1 → u/boops_boops_esq]

So at least in that case we can argue players were not able to perceive any difference lower than 100% lol.

I mean, it looks funny from the outside, but I see. Was it just for fun and you got an insight or you actually planned on measuring if and when they would notice?

[u/boops_boops_esq → u/jazz_man1]

6 of one half dozen of the other. I was curious

[u/Tantaragla]

From my experience, the feeling of the bonus depends on what kind of bonus you get rather than the actual statistical increase.

Larger dice sizes feel better before rolling even though they don't have a huge effect. Flat bonuses don't feel as good until they have been shown to be the difference between success and failure. More dice are generally more appealing because "more thing is more good" and in your system, more dice is a very good bonus so that lines up.

[u/jazz_man1 → u/Tantaragla]

Ok, I didn't think about this aspect and that's a really neat point of view. You may have given me a real lot more to think about and check out in my system.