Schervish coherence criterion

Email Notification of Changes: Click here and add TITLE of the topic to the body of the email.

Please add comments and then click on the "Add comment" button.

%COMMENT{type="belowthreadmode"}%

PeterBacchetti - 24 Jun 2012 - 00:22

To clarify and emphasize, my main point is just that if observed data x makes H more likely and makes H\x92 less likely, then x has supported H more than H\x92. And this can happen even when H implies H\x92. I used a simple, Bayesian example for clarity, but I think the problem is general. An observation x can support H more than H\x92 if it is very incompatible with (H\x92 and not H).

I believe that Jeff is mistaken when he says that x is neutral by likelihood ratio for H vs H\x92 in my example. The likelihood ratio P(x|H)/P(x|H\x92) is 2, and it cannot be calculated without using the priors (or at least the ratio P(A)/P(A or B)).

JeffreyWelge - 03 Jun 2012 - 10:10

Peter\x92s example is very interesting, and he proposes a criterion for a sensible measure of support (evidence) in terms of prior versus posterior probabilities. As these are related via the likelihood function, I think he is on the right track. A candidate measure of evidence is the likelihood ratio between two hypotheses. If we have prior information on the probabilities of A, B and C it may be helpful (and sometimes necessary) to use it \x96 but such information is not necessary in this example to describe the relative evidence conveyed by the observation x about the hypotheses considered. Suppose we do not have the priors. The probability of x under each hypothesis is sufficient: 1) Having observed x, we know that B is false since P(x|B)=0 2) Now H=H\x92=A, so the observation x is clearly neutral evidence for H v H\x92: It supports H\x92 at least as much as H, as required and as Peter noted. It is also neutral regarding the hypothesis \x93C\x94 with respect to \x93A\x94. (the probabilities of C and A have both increased by the factor 4/3 \x96 but is x evidence \x93for\x94 either in isolation? It IS definitive evidence for either over B) Regardless of the prior probabilities of the hypotheses under consideration, if we can compute the probability of x under each hypothesis, we might measure the relative evidence for H1 v H2 by the likelihood ratio P(x|H1)/P(x|H2). In Peter\x92s example, before taking the observation we would not have been able to compute P(x|H\x92) without the prior information \x96 it was fortunate that x ruled out B. The p-value fails as a measure of support because it refers to only one hypothesis, and I suggest that a sensible measure of evidence must reflect relative support for >1 hypothesis. The rule \x93if P(H) < P(H|x) then x is positive support for H\x94 conforms: Though it refers directly only to H, P(H|x) depends on the probability of x under all alternatives in ~H as well as the priors on those alternatives. If we change the example only by making P(x|B)=0.01 rather than 0, we DO need the prior odds on A v B to determine that P(x|H\x92)=0.505. The likelihood ratio now tells us that x supports H over H\x92 by a factor of nearly 2, but that seems very wrong. The problem is that the logical relationship between H and H\x92 is not encoded in the data. The Bayesian calculation shows {P(H\x92|x)=0.3355}>{P(H|x)=0.3333}. Yet as before, P(H)<P(H|x), while P(H\x92)>P(H\x92|x). Wow \x96 is Peter right? Is the search for a measure of support that will always conform to logic for this problem doomed, even if we commit fully to Bayesianism?

PeterBacchetti - 01 Jun 2012 - 10:38

Schervish proposes a “simple logical condition” that if hypothesis H implies hypothesis H’ then any observed data x must support H’ at least as much as it supports H. I believe that this is mistaken.

If H implies H’, then we do have
P(H’|x) >= P(H|x),
but these posterior probabilities are not measures of how much x has supported H or H’. I propose that such a measure should reflect how the probabilities given x differ from the prior probabilities before x was observed. Specifically,
if P(H) > P(H|x) then x has provided negative support for H, and
if P(H) < P(H|x) then x has provided positive support for H.

Even if H implies H’, it can happen that
P(H|x) > P(H) and
P(H’|x) < P(H’).
In this case, x supports H (positive support) more than it supports H’ (negative support), violating Schervish’s proposed condition, but no measure of support can meet it and still make any sense. This can happen in situations where x is fairly incompatible with the event (H’ & not H) but fairly compatible with H.

A simple example is the following:

3 possible states of nature: A, B, or C
H: A
H’: A or B
H implies H’

Prior probabilities: P(A) = 1/4 P(B) = 1/4 P(C) = 1/2
Observed data probabilities: P(x|A) = 1 P(x|B) = 0 P(x|C) = 1

Simple calculations show:

P(H|x) = P(A|x) = 1/3 > 1/4 = P(H)
Any sensible measure of support must show positive support of x for H, because P(H|x) > P(H)

P(H’|x) = P(A or B | x) = 1/3 < 1/2 = P(H’)
Any sensible measure of support must show negative support of x for H’, because P(H’|x) < P(H’).

(Note that we do still have P(H’|x) >= P(H|x), which is the condition that must hold if H implies H’.)

Therefore, no sensible measure of support can be “coherent” by the Schervish criterion. Showing that a measure violates this condition is therefore not necessarily a flaw in that measure, and any measure that meets this condition will not make sense in situations like the above.

This, of course, does not mean that P-values are sensible measures of support for the null hypothesis, but I believe that exposition based on Schervish’s criterion will not be helpful.

The distinction between the posterior probability P(H|x) and the support that x provides for H is why the analogy with the coherence criterion for multiple comparisons does not hold. If H implies H’, then it would not make sense to reject H’ but not reject H, because we must have P(H’|x) >= P(H|x). The multiple comparisons coherence criterion follows from the posterior probabilities, and does not depend on the support that x provides for H and H’.