This document discusses Bayesian variable selection methods for regression models. It begins by reviewing traditional ANOVA tables and their limitations for modern applications with many variables, such as GWAS studies. It then introduces Bayesian approaches using priors to perform variable selection by building it into the regression model. Several variable selection methods are described that use different prior distributions, such as slab and spike priors, the stochastic search variable selection (SSVS) method, and the normal-exponential-gamma (NEG) distribution. The document discusses how these methods can be implemented using MCMC sampling and compares their performance. It also discusses some extensions like using random effects and polynomial terms.

1. Bayesian Variable Selection and the
(Ab)use of Priors
Bob O'Hara
BiK-F
Frankfurt am Main
Germany
blogs.nature.com/boboh/2012/07/16/abusing_a_prior
(this is mainly a review of work by other people)

2. The Bad Old Days
ANOVA tables

3. Not Useful for Modern Applications
GWAS: 105 variables
Ikram MK et al (2010) Four Novel Loci (19q13, 6q24, 12q24, and 5q14) Influence the Microcirculation In Vivo. PLoS Genet. 2010 Oct 28;6(10):e1001184.

4. Anyway, we want to be Bayesian
Could use DIC, but same problems
So, let's build variable
selection into the model

5. Health Warnings I
I am not saying you should use variable selection

6. Health Warnings I
I am not saying you should use variable selection
p-values are EVIL

7. Health Warnings II
The methods I am about to describe are sensitive
to the priors

8. The Regression Problem
Our model:
K
y i =β0 + ∑ βk X ik +εi
k =1
(everything else is just a variant)

9. Bayesian Approach
Posterior:
K
P (β , β0, σε∣y)∝ P ( y∣β , β0, σε ) P (β0 ) P (σ ε ) ∏ P (βk )
k =1

10. Bayesian Approach
Posterior:
K
P (β ,β0, σ ε∣y)∝ P ( y∣β ,β0, σ ε ) P (β0) P (σ ε ) ∏ P (βk )
k =1
Likelihood

11. Bayesian Approach
Posterior:
K
P (β ,β0, σ ε∣y)∝ P ( y∣β ,β0, σ ε ) P (β0) P (σ ε ) ∏ P (βk )
k =1
Likelihood Priors for
regression
parameters

12. Fitting: use MCMC
Creates Markov chain
Loops through the parameters
Simply drop uninteresting parameters
marginalisation

13. The advantage (for us) of MCMC
We can over-parameterise
Some MCMC samplers (e.g Gibbs) are more
efficient
Run faster & mix better

14. The advantage (for us) of MCMC
With some imagination, we can
design priors that will work for us

15. Variable Selection
Which of the X's should be in
the model?
alternatively
Which of the β's should be zero?

16. Choosing X's
rjMCMC
General method for moving
between models with
different number of
dimensions

17. Setting βs to 0
Easier to implement
But can be slower
Stays in large dimensions

18. Slab and Spike Priors
Spike
Slab

19. Slab and Spike Posteriors
Bimodal

20. Several ways of getting priors

21. Method I: Point Mass at 0
P (β)=(1− p)0+ p N (0, σ β )

22. Indicators
Ik – indicator that variable k is in the model
P(Ik=1) = p
θ ~ N(0,σβ2)
P(β) = (1-I) 0 + I θ
And integrate over P(I=1) by MCMC
Gibbs sampling should work nicely

23. A problem with Gibbs Sampling
P(β) = (1-I) 0 + I θ
When I = 0, θ only depends on its prior
So MCMC draws wide values of θ
Only rarely will it draw
“sensible” values

24. A Better Version: GVS
θ ~ N(0,σβ2(I)) Pseudo-prior
P(β) = I θ
Now if I=0, generate from a pseudo-
prior, tuned to propose sensible
values
i.e. select σβ2(0) to cover
likely values of the posterior

25. Another way
The spike can be around 0, not
exactly on it

26. SSVS: Mixture distributions
Stochastic Search Variable Selection
Mixture of normals Spike
Slab

27. SSVS
β ~ N(0, σβ2(I)) I ~ Bern(p)
σβ2 (1)<< σβ2(0) Spike
Slab

28. Adaptive Shrinkage
Make a continuous mixture of distributions
Marginalise over the continuous mixture

29. Jeffrey's Prior
β ~ N(0, σβ2)
log(σβ2) ~ Unif(-∞,∞)

30. Bayesian Lasso
β ~ N(0, σβ2)
σβ2 ~ Exp(µ)
so β ~ dExp(µ)

31. Normal Exponential Gamma
Integrate µ from Lasso over a Gamma
β ~ N(0, σβ2)
σβ2 ~ Exp(µ)
µ ~ Γ(λ,γ 2)
NEG Exponential

32. NEG & Lasso
GWAS too big for MCMC
Use quicker algorithms & only estimate
posterior modes

33. How do they compare?
Want good
separation
Good
Bad

34. Comparison
Laplace – awful. Shrinks
everything
GVS – works well (when
tuned), but slower
SSVS – works well
Jeffrey's – works very well

35. Fixed and Random Effects
Rather than fixing parameters, can
treat as a random effect to tune it
e.g. SSVS
β ~ N(0, σI2)
σ12 ~ Γ(), σ02 = c σ12 (c<<1)
Useful with many variables, can learn about scale of
response

36. Random Effects
Useful with many variables, can learn about scale of
response
Variables not in model get P(I=1|data) = P(I=1)
Random Effect

37. Some Extensions
These might be useful sometimes
Random Effect
Variances
Polynomials

38. Random Effect Variances
Simple 1 level model
y i =α 0 +α 1 ( g i )+εi
α 1 (k )∼ N (0, σ α )
2

39. ICC
Intra-class correlation
σ 2
α
ICC = 2 2
σ α +σ

40. Variable selection on the variance
“GVS”
σα=
{ 0 I =1
exp(χ 2) I =2
“Jeffreys Shrinkage”
3, 3
“SSVS” log (σ α )∼ U (−10 10 )
σα=
{
exp(χ 1 ) I =1
exp(χ 2) I =2

41. Pr(I=1)
Pr(I=1)
True ICC

42. Estimated ICCs
Estimated ICC
True ICC

43. Polynomials
K
y i =β0 + ∑ βk X +εi
k
i
k =1
Transform to orthogonal polynomials
(use poly() in R)

44. Polynomials
K
y i =β0 + ∑ βk X i (k )+ε i
k =1
o – order of polynomial: ο ∈{1, ..., O}
βk ~ N(0, σβ2) if k ≤ o, else βk = 0

45. Why bother?
We have splines already
Polynomials are usually too wiggly

46. But...
Bayesian approaches integrate
over the parameters
will this smooth out the wiggles?

47. A test
A Response
True curve: y∝x ½
100 points
A Covariate

48. The Fitted Curve
A Response
A Covariate

49. Deviations from true mean
Deviation
A Covariate

50. Polynomial Order
Posterior Probability
Order of Polynomial

51. A couple of comments
The methods are flexible: we
can try new, weird things
Integrating over the
uncertainty smooths things

52. Does any of this make sense?
Are we abusing our priors?

53. What subjectivist priors mean
Statement about our beliefs

54. What the priors are doing here
Tuning the model to give sparseness

55. Sometimes we can bridge the gap

56. Genetics
Look at lots of loci (1000s)
Only a few are linked to genes
that have an effect
Hence, most effects are close to
zero

57. A Subjective Prior for Gene Effects?

58. What about other cases?
e.g. a recent paper
in Science

59. Response: respiration
Predictors:
Climate (PCA)
Slope, latitude, longitude etc.
Number of shrub species
Most probably have some effect
(or are correlated with something that
does)

60. A prior?
But doesn't separate variables very well

61. If we want to do variable selection...
Should first think about priors
If our subjective prior doesn't
shrink properly, either don't
select variables, or admit to
yourself you're abusing your
priors

62. Thank you for not abusing me
blogs.nature.com/boboh/2012/07/16/abusing_a_prior