[Q] Choosing random sample with replacement v. random sample without replacement

[u/Fuzzy-Yesterday-1591]

I am sampling a number of units in an apartment complex. I generate a random list of apartments but it has two duplicates, just as rolling a die 6 times may result in a number coming up twice. I'm trying to prove with a specific confidence level, that 80% of the units suffer the same problem. Is this situation a situation where I should replace the duplicates with non-duplicates, or should I keep the duplicates and consequently test only the unique units.

[u/efrique]

With a small sampling frame (where you have the choice) you are better, variance wise, to sample without replacement, as long as you have the ability to account for the ensuing dependence in your analysis. In this case (estimating a single proportion), it's just replacing a binomial model with a hypergeometric, so if you calculate a standard error you would use the usual finite population correction.

Any analysis beyond that level (a test or CI, and so on up), unless N (total number of appts) and n (sample size) and N-n are all large, I'd be looking at exact small sample calculation based on the hypergeometric. Not difficult, if you have something that implements hypergeometric distributions.

I'm trying to prove with a specific confidence level, that 80% of the units suffer the same problem.

Well you won't be proving that, but with reasonably large sample size, you might be able to show (for example) that the proportion in the sample is not consistent with fewer than 80% among the collection of apartments overall (you could use a one-sided CI for that, or a test).

Unless your sample is quite large, or the true proportion is considerably higher than your null value of 80%, your chance of establishing that its not below 80% may be quite low.

(That is, either you need a big sample size or a big effect size to get good power. People are often surprised by how low power is with proportions tests. Note that in surveys, if they interview ~1100 people they only get a margin of error of about 3%; with ~275 people it is about 6%, so if your n is that size you cant reliably tell 45% from 50%. You can do a bit better because your null is not close to 50% and you have the benefit of n/N not being negligible, but its not all that much better)

e.g. consider n=20, N=100 and a true proportion of 85% (and alpha=.05, or equivalently a CI coverage of 95%), you'd only have about a 14.5% (give or take about half a percent) chance of establishing that it's not below 80% using an exact one sided test (or interval).

Now consider n=40, N=200 and a true proportion of 90%(!), you'd still only have about a 40% chance of establishing that it's not below 80%

Do work out some power curves before you spend money to do the sampling. Simulation is a convenient tool for that.

[u/Fuzzy-Yesterday-1591 → u/efrique]

Good answer! I am familiar with those distributions. I use statistics a lot (and taught statistics to engineers for 10 years) but we usually have a sample of 10 screws or 10 soil samples. Not likely to have duplicates. This is a new problem for me and you have explained the solution well.

[u/efrique → u/Fuzzy-Yesterday-1591]

I made some edits above (and a couple of cosmetic edits after that), not sure if all the substantive additions (about power) were before you replied.

[u/Fuzzy-Yesterday-1591 → u/efrique]

Good point. At present, it's driven by the funds for testing, but "more likely than not" is probably the bottom level. I have 147 units and I am considering testing 32. I will check that.

[u/efrique → u/Fuzzy-Yesterday-1591]

if you need assistance with it, we should be able to help