This document discusses sampling-based approaches for calculating marginal densities from conditional distributions. It introduces substitution algorithms, substitution sampling, Gibbs sampling, and importance sampling. Substitution algorithms iteratively estimate marginal densities by substituting conditional distributions. Substitution sampling generates samples by iteratively drawing from conditional distributions. Gibbs sampling repeatedly draws values from conditional distributions to estimate joint and marginal distributions.

1. Introduction
Sampling Approaches
Examples
Numerical Illustrations
Conclusion
Sampling-Based Approaches to Calculating
Marginal Densities
ALAN E.GELFAND AND F.M.SMITH
Presented by Xiaolin CHENG
Reading Seminar, December 17, 2012
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

2. Introduction
Sampling Approaches
Examples
Numerical Illustrations
Conclusion
Contents
1 Introduction
2 Sampling Approaches
Substitution Algorithm
Substitution Sampling
Gibbs Sampling
Importance-Sampling Algorithm
3 Examples
4 Numerical Illustrations
5 Conclusion
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

3. Introduction
Sampling Approaches
Examples
Numerical Illustrations
Conclusion
Introduction
Abstract
The problem addressed in this paper is how to obtain numerical esti-
mates of available marginal densities, simply by means of simulated
samples from available conditional distributions, and without recourse
to sophisticated numerical analytic methods.
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

4. Introduction
Sampling Approaches
Examples
Numerical Illustrations
Conclusion
Introduction
We discuss and extend three alternative approaches put forward in
the literature for calculating marginal densities via sampling algo-
rithms.
The Substitution Algorithm
The Gibbs Sampler Algorithm
The Importance-Sampling Algorithm
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

5. Introduction
Sampling Approaches
Examples
Numerical Illustrations
Conclusion
Introduction
We discuss and extend three alternative approaches put forward in
the literature for calculating marginal densities via sampling algo-
rithms.
The Substitution Algorithm
The Gibbs Sampler Algorithm
The Importance-Sampling Algorithm
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

6. Introduction
Sampling Approaches
Examples
Numerical Illustrations
Conclusion
Introduction
We discuss and extend three alternative approaches put forward in
the literature for calculating marginal densities via sampling algo-
rithms.
The Substitution Algorithm
The Gibbs Sampler Algorithm
The Importance-Sampling Algorithm
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

7. Introduction
Sampling Approaches
Examples
Numerical Illustrations
Conclusion
Introduction
We discuss and extend three alternative approaches put forward in
the literature for calculating marginal densities via sampling algo-
rithms.
The Substitution Algorithm
The Gibbs Sampler Algorithm
The Importance-Sampling Algorithm
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

8. Introduction
Substitution Algorithm
Sampling Approaches
Substitution Sampling
Examples
Gibbs Sampling
Numerical Illustrations
Importance-Sampling Algorithm
Conclusion
Sampling Approaches
In relation to a collection of random variables,U1 , U2 , · · · , Uk ,suppose
that either
1 For i = 1, · · · , k , the conditional distributions Ui |Uj (j i ) are
available,perhaps having for some i reduced forms Ui |Uj (j ∈ Si )
(Si ⊂ {1, · · · , k })
2 The functional form of the joint density of U1 , U2 , · · · , Uk is
known and at least one Ui |Uj (j i ) is available,
Where available means that samples of Ui can be straightforwardly
and efficiently generated.
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

9. Introduction
Substitution Algorithm
Sampling Approaches
Substitution Sampling
Examples
Gibbs Sampling
Numerical Illustrations
Importance-Sampling Algorithm
Conclusion
Sampling Approaches
In relation to a collection of random variables,U1 , U2 , · · · , Uk ,suppose
that either
1 For i = 1, · · · , k , the conditional distributions Ui |Uj (j i ) are
available,perhaps having for some i reduced forms Ui |Uj (j ∈ Si )
(Si ⊂ {1, · · · , k })
2 The functional form of the joint density of U1 , U2 , · · · , Uk is
known and at least one Ui |Uj (j i ) is available,
Where available means that samples of Ui can be straightforwardly
and efficiently generated.
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

10. Introduction
Substitution Algorithm
Sampling Approaches
Substitution Sampling
Examples
Gibbs Sampling
Numerical Illustrations
Importance-Sampling Algorithm
Conclusion
Sampling Approaches
In relation to a collection of random variables,U1 , U2 , · · · , Uk ,suppose
that either
1 For i = 1, · · · , k , the conditional distributions Ui |Uj (j i ) are
available,perhaps having for some i reduced forms Ui |Uj (j ∈ Si )
(Si ⊂ {1, · · · , k })
2 The functional form of the joint density of U1 , U2 , · · · , Uk is
known and at least one Ui |Uj (j i ) is available,
Where available means that samples of Ui can be straightforwardly
and efficiently generated.
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

11. Introduction
Substitution Algorithm
Sampling Approaches
Substitution Sampling
Examples
Gibbs Sampling
Numerical Illustrations
Importance-Sampling Algorithm
Conclusion
Sampling Approaches
Densities are denoted generically by brackets and multiplication of
densities is denoted by ∗, so
The joint distribution [X , Y ]
The conditional distribution [X |Y ]
The marginal distribution [X ]
[X , Y ] = [X |Y ] ∗ [Y ]
h (Z , W ) ∗ [W ] to denote,for given Z, the expectation of the
function h (Z , W ) with respect to the marginal distribution for
W.
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

12. Introduction
Substitution Algorithm
Sampling Approaches
Substitution Sampling
Examples
Gibbs Sampling
Numerical Illustrations
Importance-Sampling Algorithm
Conclusion
Substitution Algorithm
The substitution algorithm for finding fixed-point solutions to certain
classes of integral equations is a standard mathematical tool that
has received considerable attention in the literature. Briefly review-
ing the essence of their development using the notation introduced
previously, we have
[X ] = [ X |Y ] ∗ [ Y ] (1)
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

13. Introduction
Substitution Algorithm
Sampling Approaches
Substitution Sampling
Examples
Gibbs Sampling
Numerical Illustrations
Importance-Sampling Algorithm
Conclusion
Substitution Algorithm
and
[Y ] = [Y |X ] ∗ [X ] (2)
so substituting (2) into (1) gives
[X ] = [X |Y ] ∗ [Y |X ] ∗ [X ] = h (X , X ) ∗ [X ] (3)
where h (X , X ) = [X |Y ] ∗ [Y |X ] , with X appearing as a dummy
argument in (3),and of course [X ] = [X ]
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

14. Introduction
Substitution Algorithm
Sampling Approaches
Substitution Sampling
Examples
Gibbs Sampling
Numerical Illustrations
Importance-Sampling Algorithm
Conclusion
Substitution Algorithm
Now , suppose that on the right side of (3) , [X ] were replaced by
[X ]i , to be thought of as an estimate of [X ] = [X ] arising at the ith
stage of an iterative process. Then (3) implies that
[X ]i +1 = h (X , X ) ∗ [X ]i = Ih [X ]i
where Ih is the integral operator associated with h.
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

15. Introduction
Substitution Algorithm
Sampling Approaches
Substitution Sampling
Examples
Gibbs Sampling
Numerical Illustrations
Importance-Sampling Algorithm
Conclusion
Substitution Algorithm
Exploiting standard theory of such integral operators , Tanner and
Wong ( 1987 ) showed that under mild regularity conditions this iter-
ative process has the following properties( with obviously analogous
results for( [Y ] )
The true marginal density, [X ] , is the unique solution to (3)
For almost any [X ]0 , the sequence [X ]1 , [X ]2 , . . . defined by
[X ]i +1 = Ih [X ]i (i = 0, 1, . . .) converges monotonically in L1
to [X ]
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

16. Introduction
Substitution Algorithm
Sampling Approaches
Substitution Sampling
Examples
Gibbs Sampling
Numerical Illustrations
Importance-Sampling Algorithm
Conclusion
Substitution Algorithm
Extending the substitution algorithm to three random variables X, Y,
and Z , we may write [ analogous to (1) and (2) ]
[X ] = [X , Z |Y ] ∗ [Y ] (4)
[Y ] = [Y , X |Z ] ∗ [Z ] (5)
and
[Z ] = [ Z , Y |X ] ∗ [ X ] (6)
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

17. Introduction
Substitution Algorithm
Sampling Approaches
Substitution Sampling
Examples
Gibbs Sampling
Numerical Illustrations
Importance-Sampling Algorithm
Conclusion
Substitution Algorithm
Substitution of (6) into (5) and then (5) into (4) produces a fixed-
point equation analogous to (3). A new h function arises with asso-
ciated integral operator Ih , and these properties continue to hold in
this extended setting. Extension to k variables is straightforward.
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

18. Introduction
Substitution Algorithm
Sampling Approaches
Substitution Sampling
Examples
Gibbs Sampling
Numerical Illustrations
Importance-Sampling Algorithm
Conclusion
Substitution Sampling
Returning to (1) and (2) , suppose that [X |Y ] and [Y |X ] are available
in the sense defined at the beginning.
For an arbitrary initial marginal distribution [X ]0 draw a single
distribution X 0 from [X ]0
Given X 0 , since [Y |X ] is available draw Y (1) ∼ [Y |X (0) ], and
hence from (2) the marginal distribution of [Y (1) ] is [Y ]1 =
[Y |X ] ∗ [X ]0
Now,complete a cycle by drawing X (1) ∼ [X |Y (1) ]. Using (1),
we then have X (1) ∼ [X ]1 = [X |Y ] ∗ [Y ]1 = h (X , X ) ∗
[X ]0 = Ih [X ]0
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

19. Introduction
Substitution Algorithm
Sampling Approaches
Substitution Sampling
Examples
Gibbs Sampling
Numerical Illustrations
Importance-Sampling Algorithm
Conclusion
Substitution Sampling
Repetition of this cycle produces Y (2) and X (2) , and eventually, af-
ter i iterations, the pair (X (i ) , Y (i ) ) such that X (i ) → X ∼ [X ], and
Y (i ) → Y ∼ [Y ]. Repetition of this sequence m times each to the
(i ) (i )
ith iteration generates m iid pairs (Xj , Yj ) (j = 1, . . . , m). We call
this generation scheme substitution sampling.
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

20. Introduction
Substitution Algorithm
Sampling Approaches
Substitution Sampling
Examples
Gibbs Sampling
Numerical Illustrations
Importance-Sampling Algorithm
Conclusion
Substitution Sampling
If we terminate all repetitions at the ith iteration, the proposed den-
sity estimate of [X ] (with an analogous expression for [Y ] ) is the
Monte Carlo integration
m
ˆ 1 (i )
[X ]i = [X |Yj ] (7)
m
j =1
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

21. Introduction
Substitution Algorithm
Sampling Approaches
Substitution Sampling
Examples
Gibbs Sampling
Numerical Illustrations
Importance-Sampling Algorithm
Conclusion
Substitution Sampling
Extension of the substitution-sampling algorithm to more than two
random variables is straightforward. We illustrate using the three-
variable case.Paralleling (7), the density estimator of [X] becomes
m
1 (i ) (i )
ˆ
[X ]i = [X |Yj , Zj ] (8)
m
j =1
with analogous expressions for estimating [Y] and [Z].
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

22. Introduction
Substitution Algorithm
Sampling Approaches
Substitution Sampling
Examples
Gibbs Sampling
Numerical Illustrations
Importance-Sampling Algorithm
Conclusion
Substitution Sampling
For k variables, U1 , . . . , Uk , the density estimator for [Us ](s = 1, . . . , k )
is
m
ˆs ]i = 1
[U
(i )
[Us |Ut = Utj ; t s ] (9)
m
j =1
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

23. Introduction
Substitution Algorithm
Sampling Approaches
Substitution Sampling
Examples
Gibbs Sampling
Numerical Illustrations
Importance-Sampling Algorithm
Conclusion
Gibbs Sampling
The Gibbs sampler has mainly been applied in the context of com-
plex stochastic models involving very large numbers of variables,
such as image reconstruction, neural networks, and expert system.
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

24. Introduction
Substitution Algorithm
Sampling Approaches
Substitution Sampling
Examples
Gibbs Sampling
Numerical Illustrations
Importance-Sampling Algorithm
Conclusion
Gibbs Sampling
Algorithm
(0) (0) (0)
Given an arbitrary starting set of values U1 , U2 , . . . , Uk
(1) (0) (0)
U1 ∼ [U1 |U2 , . . . , Uk ]
(1) (1) (0) (0)
U2 ∼ [U2 |U1 , U3 . . . , Uk ]
(1) (1) (1) (0) (0)
U3 ∼ [U3 |U1 , U2 , U4 , . . . , Uk ]
.
.
.
(1) (1) (1)
Uk ∼ [Uk |U1 , . . . , Uk −1 ]
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

25. Introduction
Substitution Algorithm
Sampling Approaches
Substitution Sampling
Examples
Gibbs Sampling
Numerical Illustrations
Importance-Sampling Algorithm
Conclusion
Gibbs Sampling
(i ) (i ) (i )
After i such iterations we would arrive at U1 , U2 , . . . , Uk and we
have the following results
(i ) (i ) (i ) (i )
(U1 , U2 , . . . , Uk ) → [U1 , . . . , Uk ] and hence for each s, Us →
Us ∼ [Us ] as i → ∞.
Using the sup norm, rather than the L1 norm, the joint density
(i ) (i ) (i )
of (U1 , U2 , . . . , Uk ) converges to the true joint density
For any measurable function T of U1 , . . . , Uk whose expecta-
tion exists
i
1 (l ) (l )
lim T (U1 , . . . , Uk ) → E (T (U1 , . . . , Uk ))
j →∞ i
l =1
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

26. Introduction
Substitution Algorithm
Sampling Approaches
Substitution Sampling
Examples
Gibbs Sampling
Numerical Illustrations
Importance-Sampling Algorithm
Conclusion
Importance-Sampling Algorithm
Rubin(1987) suggested a noniterative Monte Carlo method for gen-
erating marginal distributions using importance-sampling ideas and
We first present the basic idea in the two-variable case
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

27. Introduction
Substitution Algorithm
Sampling Approaches
Substitution Sampling
Examples
Gibbs Sampling
Numerical Illustrations
Importance-Sampling Algorithm
Conclusion
Importance-Sampling Algorithm
suppose that
We seek the marginal distribution of X, given only the func-
tional form of the joint density [X , Y ] and the availability of the
conditional distribution [X |Y ]
The marginal distribution of Y is not known
Choose an importance-sampling distribution for Y that has pos-
itive support wherever [Y ] does and that has density [Y ]s
Then [X |Y ] ∗ [Y ]s provides an importance-sampling distribution for
(X , Y ).
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

28. Introduction
Substitution Algorithm
Sampling Approaches
Substitution Sampling
Examples
Gibbs Sampling
Numerical Illustrations
Importance-Sampling Algorithm
Conclusion
Importance-Sampling Algorithm
We draw iid pairs (Xl , Yl ) (l = 1, . . . , N ) from this joint distribution,
for example, by drawing Yl from [Y ]s and Xl from [X |Yl ]. Rubin’s
idea is to calculate rl = [Xl , Yl ]/[Xl |Yl ] ∗ [Yl ]s (l = 1, . . . , N ) and then
estimate the marginal density for [X ] by
N N
ˆ
[X ] = [X |Yl ]rl / rl (10)
l =1 i =1
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

29. Introduction
Substitution Algorithm
Sampling Approaches
Substitution Sampling
Examples
Gibbs Sampling
Numerical Illustrations
Importance-Sampling Algorithm
Conclusion
Importance-Sampling Algorithm
ˆ
[X ] → [X ] with probability 1 as N → ∞ for almost every X. In ad-
dition, if [Y |X ] is available we immediately have an estimate for the
marginal distribution of Y : [Y ] = N 1 [Y |Xl ]rl / N 1 rl
ˆ
l= l=
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

30. Introduction
Substitution Algorithm
Sampling Approaches
Substitution Sampling
Examples
Gibbs Sampling
Numerical Illustrations
Importance-Sampling Algorithm
Conclusion
Importance-Sampling Algorithm
The extension of the Rubin importance-sampling idea to the case
of k variables is clear. For instance, when k = 3, suppose that
we seek the marginal distribution of X, given the functional form
of [X , Y , Z ] and the availability of the full conditional [X |Y , Z ]. In
this case, the pair (Y , Z ) plays the role of Y in the two-variable case
discussed before, and in general we need to specify an importance-
sampling distribution [Y , Z ]s .
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

31. Introduction
Substitution Algorithm
Sampling Approaches
Substitution Sampling
Examples
Gibbs Sampling
Numerical Illustrations
Importance-Sampling Algorithm
Conclusion
Importance-Sampling Algorithm
We draw iid triples (Xl , Yl , Zl ) (l = 1, . . . , N ) and calculate rl =
[Xl , Yl , Zl ]/([Xl |Yl , Zl ] ∗ [Yl , Zl ]s ). The marginal density estimate for
[X ]
N N
ˆ
[X ] = [X |Yl , Zl ]rl / rl (11)
l =1 l =1
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

32. Introduction
Sampling Approaches
Examples
Numerical Illustrations
Conclusion
Examples
A major area of potential application of the methodology we have
been discussed is in the calculation of marginal posterior densities
within a bayesian inference framework. In recent years, there have
been many advances in numerical and analytic approximation tech-
niques for such calculations, but implementation of these approach-
es typically requires sophisticated numerical analytic expertise. By
contrast, the sampling approaches we have discussed are straight-
forward to implement.
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

33. Introduction
Sampling Approaches
Examples
Numerical Illustrations
Conclusion
Examples
Consider a general Bayesian hierarchical model having k stages. In
an obvious notation, we write the joint distribution of the data and
parameters as
[Y |θ1 ] ∗ [θ1 |θ2 ] ∗ [θ2 |θ3 ] ∗ · · · ∗ [θk −1 |θk ] ∗ [θk ] (12)
where we assume all components of prior specification to be avail-
able for sampling. Primary interest is usually in the marginal poste-
rior [θ1 |Y ].
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

34. Introduction
Sampling Approaches
Examples
Numerical Illustrations
Conclusion
Examples
As a concrete illustration, consider an exchangeable poisson mod-
el. Suppose that we observe independent counts, si , over differing
lengths of time, ti (with resultant rate ρi = si /ti ) (i = 1, . . . , p ).
Assume [si |λi ] = P0 (λi ti ) and that the λi are iid from G (α, β) with
density λα−1 e −λi /β /βα Γ(α)
i
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

35. Introduction
Sampling Approaches
Examples
Numerical Illustrations
Conclusion
Examples
The parameter α is assumed known (in practice, we might treat α
as a tuning parameter, or perhaps, in an empirical Bayes spirit, es-
timate it from the marginal distribution of the si s), and β is assumed
to arise from an inverse gamma distribution IG (γ, δ) with density
δγ e −δ/β Γ(γ)
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

36. Introduction
Sampling Approaches
Examples
Numerical Illustrations
Conclusion
Examples
Letting Y = (s1 , . . . , sp ), the conditional distributions[λj |Y ] are sought.
The full conditional distribution of λj is given by
[λj |Y , β, λi ,i j ] = G (α + sj , (tj + 1/β)−1 ) (13)
whereas the full conditional distribution for β is given by
[β|Y , λ1 , . . . , λp ] = IG (γ + p α, λi + δ) (14)
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

37. Introduction
Sampling Approaches
Examples
Numerical Illustrations
Conclusion
Examples
(0) (0) (0) (1)
Given (λ1 , λ2 , . . . , λp , β(0) ) the Gibbs sampler draw λj ∼ G (α+
(1)
sj , (tj + 1/β(0) )−1 ) (j = 1, . . . , p ) and β(1) ∼ IG (γ + p α, λi + δ) to
(i ) (i ) (i ) (i )
complete one cycle, generating (λ1l , λ2l , . . . , λpl , βl )(l = 1, . . . , m)
the marginal density estimate for λj is
1 1
[λjˆY ] =
| G (α + sj , (tj +
(i )
)−1 )(j = 1, . . . , p ) (15)
m βl
whereas
m
1
[β|Y ] =
ˆ IG (γ + αp , λijl + δ)) (16)
m
l =1
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

38. Introduction
Sampling Approaches
Examples
Numerical Illustrations
Conclusion
Examples
Rubin’s importance-sampling algorithm is applicable in the setting
(12) as well, taking a particularly simple form in the case k =
2, 3. For k = 3, suppose that we seek [θ1 |Y ]. The joint density
[θ1 , θ2 , θ3 |Y ] = [Y , θ1 , θ2 , θ3 ]/[Y ], where the functional form of the
numerator is given in (12).
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

39. Introduction
Sampling Approaches
Examples
Numerical Illustrations
Conclusion
Examples
An importance-sampling density for [θ1 , θ2 , θ3 |Y ] could be sampled
as [θ1 |Y , θ2 ] ∗ [θ3 |θ2 ] ∗ [θ2 |Y ]s for some [θ2 |Y ]s . A good choice for
[θ2 |Y ]s might be obtained through a few iterations of the substitution-
sampling algorithm. In any case, for l = 1, . . . , N we would generate
θ2l from [θ2 |Y ]s , θ3l from [θ3 |θ2l ], and θ1l from [θ1 |Y , θ2l ]. Calculating
[Y , θ1l , θ2l , θ3l ]
rl =
[θ1l |Y , θ2l ] ∗ [θ3l |θ2l ] ∗ [θ2l |Y ]s
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

40. Introduction
Sampling Approaches
Examples
Numerical Illustrations
Conclusion
Examples
We obtain the density estimator
[θ1ˆY ] =
| [θ1 |Y , θ2l ]rl / rl
Returning to the exchangeable Poisson model, the estimator of the
marginal density of λj under rubin’s importance-sampling algotithm
is
N N
1
[λjˆY ] =
| G (α + sj , (tj + )−1 )rl / rl (17)
l =1
βj l =1
where rl = [Y |βl ] ∗ [βl ]/[βl |Y ]s
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

41. Introduction
Sampling Approaches
Examples
Numerical Illustrations
Conclusion
Numerical Illustrations
We apply the exchangeable Poisson model to data on pump fail-
ures, where si is the number of failures and ti is the length of time
in thousands of hours.
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

42. Introduction
Sampling Approaches
Examples
Numerical Illustrations
Conclusion
Pump system si ti ρi (×102 )
1 5 94.320 5.3
2 1 15.720 6.4
3 5 62.880 8.0
4 14 125.760 11.1
5 3 5.240 57.3
6 19 31.440 60.4
7 1 1.048 95.4
8 1 1.048 95.4
9 4 2.096 191.0
10 22 10.480 209.9
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

43. Introduction
Sampling Approaches
Examples
Numerical Illustrations
Conclusion
Numerical Illustrations
Recalling the model structure and the forms of conditional distri-
bution given by (13) and (14), we illustrate the use of the Gibbs
sampler for this data set, with p = 10, δ = 1, γ = 0.1, and for the
¯ ¯ p
purposes of illustration α = ρ2 /(Vρ − ρ−1 ρ i =1 ti−1 ) Where ρ = α/β
¯
and Vρ = p −1 (ρ − ρ)2 ¯
i
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

44. Introduction
Sampling Approaches
Examples
Numerical Illustrations
Conclusion
Numerical Illustrations
The cycle is defined as follows:
draw initial β0 from [β]. where β ∼ IG (γ, δ)
( 1)
draw independent λj from [λj |Y , β(0) , λj , j i ]. which is a
G (α + sj , (tj + 1
β(0)
) −1 ) distribution, j = 1, . . . , p
(1) (1)
draw β(1) from [β|Y , λ1 , . . . , λp ]. which is an IG (γ + αp , δ +
(1)
λi ) distribution.
Reinitialize the cycle with β1 and iterate, replicating each cycle m
times.
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities

45. Introduction
Sampling Approaches
Examples
Numerical Illustrations
Conclusion
Conclusion
We have emphasized providing a comparative review and explica-
tion of three possible sampling approaches to the calculation of in-
tractable marginal densities. The substitution, Gibbs, and importance-
sampling algorithms are all straightforward to implement in several
frequently occurring practical situation, thus avoiding complicated
numerical or analytic approximation exercises.
Xiaolin CHENG Sampling-Based Approaches to Calculating Marginal Densities