Probability of direction

In Bayesian statistics, the probability of direction (pd) is a measure of effect existence representing the certainty with which an effect is positive or negative.^[1] This index is numerically similar to the frequentist p-value.^[2]^[3]

Definition

It is mathematically defined as the larger of two posterior probabilities: the probability of the parameter ( $\theta$ ) being negative and the probability of the parameter being positive: $pd={\text{max}}\{Pr(\theta <0),Pr(\theta >0)\}$ When the posterior is a continuous distribution, this value is also equal the proportion of the posterior distribution that is of the median's sign, and varies between 50% and 100%.^[4] However, when the posterior is a discrete distribution (or a mixture of continuous and discrete values), this value might not be equal to the proportion of the posterior distribution that is of the median's sign (as the median might be equal to the null value), and can be as low as 0% (when the null has a 100% posterior probability).

History

The original formulation of this index and its usage in Bayesian statistics can be found in the psycho software documentation by Dominique Makowski under the appellation Maximum Probability of Effect (MPE).^[5]^[6] It was later renamed Probability of Direction and implemented in the easystats collection of software. Similar formulations have also been described in the context of bootstrapped parameters interpretation.^{[citation needed]}

Properties

The probability of direction is easy to estimate, is typically independent of the specific parameterization of the model or scaling of the predictor(s) or outcome, and is not sensitive to prior specification (insofar as the posterior is not prior specification).

Advantages and limitations of the probability of direction have been studied by comparing it to other indices including the Bayes factor or Bayesian Equivalence test:^[2]^[7]^[4]^[8] Unlike indices related to the Region of Practical Interest (ROPE; where rescaling a variable requires rescaling the ROPE), its computation is not sensitive to the scale of the response or predictor variables. Unlike the Bayes factor - an index of relative fit of the prior model - the probability of direction it is an index of the posterior distribution. Like the ROPE and unlike the Bayes factor, the probability of direction can easily be estimated from MCMC samples by counting the proportion of samples that are larger than the null and the proportion of samples smaller than the null, and taking the larger of the two.

However, similarly to its frequentist counterpart - the p-value - this index is not able to quantify evidence in favor of the null hypothesis since in most applied cases it cannot be lower than 50% (see above).^[2]^[9]

Relationship with p-value

In cases that produce posterior distributions that are close in shape to frequentists sampling distributions (such as the conditions required for the Bernstein–von Mises theorem) the probability of direction has a direct correspondence with the frequentist one-sided p-value through the formula $p_{\text{one-sided}}=1-pd$ and to the two-sided p-value through the formula $p_{\text{two-sided}}=2\left(1-pd\right)$ . Thus, a two-sided p-value of respectively .1, .05, .01 and .001 would correspond approximately to a pd of 95%, 97.5%, 99.5% and 99.95%.^[10] The proximity between the pd and the p-value is in line with the interpretation of the former as an index of effect existence, as it follows the original definition of the p-value.^[11]^[12]

Interpretation

The probability of direction can be interpreted as a continuous measure of evidence, without "testing" it against a fixed threshold.

However, for instances where a decision is required, the bayestestR package for R suggests the following rule of thumb guidelines, based on the association between it and the frequentist p-value:^[13]

pd	p-value equivalence	Interpretation
$\leq 95\%$	$p>.1$	Uncertain
$>95\%$	$p<.1$	Possibly existing
$>97\%$	$p<.06$	Likely existing
$>99\%$	$p<.02$	Probably existing
$>99.9\%$	$p<.002$	Certainly existing