Inappropriate Statistical Testing for Confounding

Lead Author(s): Jeff Martin, MD

Examining a Known Confounder

Testing for statistically significant differences (adjustment) between crude and adjusted measures is inappropriate when:

Factor is a known confounder. Why don’t we just apply a statistical test to see if the difference between the crude and adjusted measure of association is significant? This is a bad idea. Even if you wanted to perform a statistical test, they are actually not available in software packages.

Small Sample Size

Testing for statistically significant differences (adjustment) between crude and adjusted measures is inappropriate when:

The study has a small sample size, even large differences between crude and adjusted measures may not be statistically different. Why is statistical testing for confounding inappropriate? What if you happened to have a small sample size in your study? You cannot ignore differences between crude and adjusted measures just because they are not statistically significant.

Issue of Confounding Is Bias

The issue of confounding is one of bias, not of sampling error. Caveat: there may be a penalty in statistical precision when controlling for potential confounders. See the study of spermicide use and Down Syndrome.

Do Not Adjust

If you adjust for a factor that you don't need to adjust for, you will end up with much larger confidence intervals and a loss in precision. See Lung Cancer and Matches Study.

Bias-Variance Tradeoff