Inappropriate Statistical Testing for Confounding
Lead Author(s): Jeff Martin, MD
Examining a Known Confounder
Testing for statistically significant differences (adjustment) between crude and adjusted measures is inappropriate when:
Factor is a known confounder.
- e.g., examining an association for which a factor is a known confounder
- age in the association between hypertension (HTN) and coronary artery disease (CAD)
Why don’t we just apply a statistical test to see if the difference between the crude and adjusted measure of association is significant? This is a bad idea. Even if you wanted to perform a statistical test, they are actually not available in software packages.
Small Sample Size
Testing for statistically significant differences (adjustment) between crude and adjusted measures is inappropriate when:
The study has a small sample size, even large differences between crude and adjusted measures may not be statistically different.
- Yet, we know confounding is present.
- Therefore, the difference between crude and adjusted measures cannot be ignored as merely chance.
- Bias must be prevented: the difference must be reported as confounding.
Why is statistical testing for confounding inappropriate?
- Say you are working with a variable that you know is a confounder in every prior instance it was examined, like age in the evaluation of the relationship between hypertension and coronary artery disease.
What if you happened to have a small sample size in your study?
- Then even large differences between the crude and adjusted measures of association might not be statistically significant.
- What are you going to do?
- Throw out age and just go with crude estimate of the relationship between hypertension and CAD?
- Of course, you won’t do this because you know that confounding is present!
You cannot ignore differences between crude and adjusted measures just because they are not statistically significant.
- In other words, when protecting against bias, you have to do whatever you can regardless of statistical significance.
- We have to live with what we see as differences between crude and adjusted regardless of the statistics.
Issue of Confounding Is Bias
The issue of confounding is one of bias, not of sampling error.
- We must live with whatever effects we see after adjustment for a factor for which there is a strong a priori belief about confounding.
- We’re not concerned that sampling error is causing confounding and therefore we don’t have to worry about testing for role of chance
- The exception to this are variables for which we are not sure a priori are confounders. For such variables, we need to weigh bias versus variance tradeoffs.
Caveat: there may be a penalty in statistical precision when controlling for potential confounders. See the study of spermicide use and Down Syndrome.
Do Not Adjust
If you adjust for a factor that you don't need to adjust for, you will end up with much larger confidence intervals and a loss in precision. See Lung Cancer and Matches Study.
Bias-Variance Tradeoff