Chi Square versus Fisher's Exact

FORUM QUESTION: Julia St George - 22 May 2009 - 12:00

Can you explain the difference between Fisher's exact test and the chi-square test for a 2x2 table? When do you use the exact test, and how would the results differ?

FORUM ANSWER 1: FrankHarrell - 22 May 2009 - 14:53

"Fisher's exact test" is somewhat of a misnomer. It is a test for comparing two proportions that is conditional on the marginal frequencies. The test is exact in guaranteeing that the type I error is less than or equal to what you specify. It can however be much less, i.e., P-values are too large and power is lost. In this sense ordinary chi-square tests are generally "more accurate" than "exact" tests. In addition, the old adage that chi-square tests are not accurate if an expected cell frequency is less than 5 is not correct.

To me the definitive paper on the subject is Lydersen et al Stat in Med 28:1159; 2009, which states that Fisher's exact test should never be used unless the mid-P correction is applied. This paper points out the advantages of newer unconditional tests.

FORUM ANSWER 2: __-- Charity Moore, PhD, MSPH__Julia St George - 27 May 2009 - 10:49

Fisher's Exact test is a way to test the association between two categorical variables when you have small cell sizes (expected values less than 5). Chi-square test is used when the cell sizes are expected to be large. If your sample size is small (or you have expected cell sizes <5), you should use Fisher's Exact test. Otherwise, the two tests will give relatively the same answers. With large cell sizes, their answer should be very similar.

Also see:

http://www.childrensmercy.org/stats/weblog2007/DifferencesBetweenTests.asp

http://www.childrensmercy.org/stats/ask/fishers.asp

FORUM ANSWER 3: PeterBacchetti - 03 Jun 2009 - 15:19

We seem to have a direct contradiction below between, “If your sample size is small (or you have expected cell sizes <5), you should use Fisher's Exact test,” from Charity Moore on May 27 versus, “ordinary chi-square tests are generally ‘more accurate’ than ‘exact’ tests” and “the old adage that chi-square tests are not accurate if an expected cell frequency is less than 5 is not correct,” from Frank Harrell on May 22.

This may reflect a schism in the wider field of statistics, although my understanding is that Fisher’s exact test (and other exact methods) are known to be too conservative. Some regard conservatism as a good property, but really it is a form of bias and accuracy is a better goal. So I believe that avoiding Fisher’s exact test is the best recommendation, even if this means using the ordinary chi-square test.

FORUM ANSWER 4: KnutWittkowski - 07 Jun 2009 - 08:46

I agree with Frank and Peter that "exact" tests are often less appropriate than asymptotic tests, in particular when the data are not "exact", i.e., rounded or discrete surrogates for continuous concepts (e.g, parity for fertility). In the latter case, "exact" tests may not even be conservative, because the results are "conditioned on the discretization", while the investigator may be rather interested in the phenomenon before the discretization took place. Hence, "exact" tests should be avoided, in particular, when the ties are not exact, because "exact" tests may give the exact answer to the wrong question, rather than an approximate answer to the right question. Wittkowski (1998, Biometrics 54:789–791), Randles (2001, The American Statistician 55:96-101).