The document describes several bootstrap methods for estimating parameters from sample data when the underlying distribution is unknown. It outlines the bootstrap procedure, which involves resampling the original data with replacement to create bootstrap samples and estimating the parameter from each resample. Three methods for calculating the bootstrap distribution are described: direct theoretical calculation, simulation-based resampling, and Bayesian approaches. The document also provides an example of using the bootstrap to estimate the median from a sample.
1. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
Bootstrap Methods:
Another Look at the Jackknife
Marco Brandi
TSI-EuroBayes Student
University Paris Dauphine
26 November 2012 / Reading Seminar on Classics
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
2. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
"To pull oneself up by one is bootstrap"
Rudolph Erich Raspe
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
3. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
OUTLINE
1 INTRODUCTION
2 DESCRIPTION OF METHODS
METHOD 1
METHOD 2
METHOD 3
3 BOOTSTRAP IN REGRESSION MODELS
4 BAYESIAN BOOTSTRAP
5 DISCUSSION
6 BAG OF LITTLE BOOTSTRAP
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
4. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
Outline
1 INTRODUCTION
2 DESCRIPTION OF METHODS
METHOD 1
METHOD 2
METHOD 3
3 BOOTSTRAP IN REGRESSION MODELS
4 BAYESIAN BOOTSTRAP
5 DISCUSSION
6 BAG OF LITTLE BOOTSTRAP
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
5. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
PRESENTING THE PROBLEM
X = (X1 , . . . , Xn )
Xi ∼ F with F completely unspecified
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
6. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
PRESENTING THE PROBLEM
X = (X1 , . . . , Xn )
Xi ∼ F with F completely unspecified
GOAL
⇓
Given R(X, F ) estimate R on the basis of x = (x1 , . . . , xn )
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
7. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
INTRODUCTION JACKKNIFE METHOD
θ(F ) parameter of interest and t(X) its estimator
R(X, F ) = t(X) − θ(F )
ˆ
t(X)−Bias(t)−θ(F )
R(X, F ) =
ˆ(t))1/2
(Var
ˆ ˆ
Bias(t) and Var (t) are obtained recomputing t(·) n times , each
time removing one component of X
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
8. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
BOOTSTRAP METHOD
BOOTSTRAP METHOD
at x1 , x2 , . . . , xn put mass 1/n
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
9. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
BOOTSTRAP METHOD
BOOTSTRAP METHOD
at x1 , x2 , . . . , xn put mass 1/n
ˆ
F is the sample probability distribution
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
10. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
BOOTSTRAP METHOD
BOOTSTRAP METHOD
at x1 , x2 , . . . , xn put mass 1/n
ˆ
F is the sample probability distribution
Xi∗ = xi∗ ˆ
Xi∗ ∼ F i = 1, . . . , n
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
11. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
BOOTSTRAP METHOD
BOOTSTRAP METHOD
at x1 , x2 , . . . , xn put mass 1/n
ˆ
F is the sample probability distribution
Xi∗ = xi∗ ˆ
Xi∗ ∼ F i = 1, . . . , n
X∗ boostrap sample
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
12. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
BOOTSTRAP METHOD
BOOTSTRAP METHOD
at x1 , x2 , . . . , xn put mass 1/n
ˆ
F is the sample probability distribution
Xi∗ = xi∗ ˆ
Xi∗ ∼ F i = 1, . . . , n
X∗ boostrap sample
R∗ ˆ
= R(X∗ , F )
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
13. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
SIMPLE EXAMPLE
Dichotomous Example
θ(F ) = Pr {X = 1} ¯
R(X, F ) = X − θ(F )
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
14. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
SIMPLE EXAMPLE
Dichotomous Example
θ(F ) = Pr {X = 1} ¯
R(X, F ) = X − θ(F )
ˆ
Xi∗ = 1 x = θ(F )
¯
Xi∗ =0 1−x ¯
⇓
ˆ ¯
R ∗ = R(X∗ , F ) = X ∗ − x
¯
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
15. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
SIMPLE EXAMPLE
Dichotomous Example
θ(F ) = Pr {X = 1} ¯
R(X, F ) = X − θ(F )
ˆ
Xi∗ = 1 x = θ(F )
¯
Xi∗ =0 1−x ¯
⇓
ˆ ¯
R ∗ = R(X∗ , F ) = X ∗ − x
¯
¯
E∗ (X ∗ − x ) = 0
¯ ¯
Var∗ (X ∗ − x ) = x (1 − x )/n
¯ ¯ ¯
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
16. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
PROBLEM
The complexity on the bootstrap procedure is to calculate
the bootstrap distribution
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
17. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
PROBLEM
The complexity on the bootstrap procedure is to calculate
the bootstrap distribution
⇓
3 methods of calculation are possible
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
18. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
Outline
1 INTRODUCTION
2 DESCRIPTION OF METHODS
METHOD 1
METHOD 2
METHOD 3
3 BOOTSTRAP IN REGRESSION MODELS
4 BAYESIAN BOOTSTRAP
5 DISCUSSION
6 BAG OF LITTLE BOOTSTRAP
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
19. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
Outline
1 INTRODUCTION
2 DESCRIPTION OF METHODS
METHOD 1
METHOD 2
METHOD 3
3 BOOTSTRAP IN REGRESSION MODELS
4 BAYESIAN BOOTSTRAP
5 DISCUSSION
6 BAG OF LITTLE BOOTSTRAP
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
20. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
Method 1
Direct theoretical calculation
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
21. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
ESTIMATING THE MEDIAN 1ST STEP
Initializing the procedure
θ(F ) indicate the median of F
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
22. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
ESTIMATING THE MEDIAN 1ST STEP
Initializing the procedure
θ(F ) indicate the median of F
t(X) = X(m)
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
23. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
ESTIMATING THE MEDIAN 1ST STEP
Initializing the procedure
θ(F ) indicate the median of F
t(X) = X(m)
X(1) ≤ X(2) ≤ · · · ≤ X(n) n = 2m − 1
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
24. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
ESTIMATING THE MEDIAN 1ST STEP
Initializing the procedure
θ(F ) indicate the median of F
t(X) = X(m)
X(1) ≤ X(2) ≤ · · · ≤ X(n) n = 2m − 1
R(X, F ) = t(X) − θ(F )
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
25. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
ESTIMATING THE MEDIAN 2ST STEP
Formalazing the procedure
X∗ = x∗
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
26. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
ESTIMATING THE MEDIAN 2ST STEP
Formalazing the procedure
X∗ = x∗
Ni∗ = #{Xi∗ = xi } N∗ = (N1 , N1 , . . . .Nn )
∗ ∗ ∗
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
27. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
ESTIMATING THE MEDIAN 2ST STEP
Formalazing the procedure
X∗ = x∗
Ni∗ = #{Xi∗ = xi } N∗ = (N1 , N1 , . . . .Nn )
∗ ∗ ∗
R∗ ˆ
= R(X∗ , F ) = X(m) − x(m)
∗
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
28. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
ESTIMATING THE MEDIAN 2ST STEP
Formalazing the procedure
X∗ = x∗
Ni∗ = #{Xi∗ = xi } N∗ = (N1 , N1 , . . . .Nn )
∗ ∗ ∗
R∗ ˆ
= R(X∗ , F ) = X(m) − x(m)
∗
l −1
Pr∗ {R ∗ = x(l) − x(m) } =Pr {Bin(n, ) ≤ m − 1}−
n (1)
l
−Pr {Bin(n, ) ≤ m − 1}
n
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
29. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
RESULTS(1)
for n = 15 and m = 8
l 2 or 14 3 or 13 4 or 12 5 or 11 6 or 10 7 or 9 8
(1) .0003 .0040 .0212 .0627 .1249 .1832 .2073
15
Use E∗ (R ∗ )2 = l=1 [x(l) − x(8) ]2 Pr∗ R ∗ = x(l) − x(8)
as an estimate of EF R 2 = EF [t(X) − θ(F )]2
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
30. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
RESULTS(2)
Results for bootstrap
limn→∞ nE∗ (R ∗ )2 = 1/4f 2 (θ)
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
31. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
RESULTS(2)
Results for bootstrap
limn→∞ nE∗ (R ∗ )2 = 1/4f 2 (θ)
Results for the standard jackknife
2
limn→∞ nVarˆ(R) = (1/4f 2 (θ)) χ2
2
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
32. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
Outline
1 INTRODUCTION
2 DESCRIPTION OF METHODS
METHOD 1
METHOD 2
METHOD 3
3 BOOTSTRAP IN REGRESSION MODELS
4 BAYESIAN BOOTSTRAP
5 DISCUSSION
6 BAG OF LITTLE BOOTSTRAP
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
33. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
METHOD 2 - MONTE CARLO APPROXIMATION
Repeat X∗ B times
x∗1 , x∗2 , . . . , x∗B
ˆ ˆ ˆ
R(x∗1 , F ), R(x∗2 , F ), . . . , R(x∗B , F )
is taken as an approximation of the boostrap distribution
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
34. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
EXAMPLE(1)
Xi ∼ Pois(2) i = 1, . . . , 15
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
35. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
EXAMPLE(1)
Xi ∼ Pois(2) i = 1, . . . , 15
Histogram of bootstrap mean t(X) = E [X]
0.8
Density
0.4
0.0
0.5 1.0 1.5 2.0 2.5 3.0 3.5
Bootstrap estimation of mean
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
36. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
EXAMPLE(1)
Xi ∼ Pois(2) i = 1, . . . , 15
Histogram of bootstrap mean t(X) = E [X]
B = 10000
0.8
n◦ of bootstrap samples
Density
0.4
0.0
0.5 1.0 1.5 2.0 2.5 3.0 3.5
Bootstrap estimation of mean
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
37. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
EXAMPLE(1)
Xi ∼ Pois(2) i = 1, . . . , 15
Histogram of bootstrap mean t(X) = E [X]
B = 10000
0.8
n◦ of bootstrap samples
Density
mean = 1.9341
0.4
0.0
0.5 1.0 1.5 2.0 2.5 3.0 3.5
Bootstrap estimation of mean
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
38. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
EXAMPLE(1)
Xi ∼ Pois(2) i = 1, . . . , 15
Histogram of bootstrap mean t(X) = E [X]
B = 10000
0.8
n◦ of bootstrap samples
Density
mean = 1.9341
0.4
se = 0.382
0.0
0.5 1.0 1.5 2.0 2.5 3.0 3.5
Bootstrap estimation of mean
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
39. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
EXAMPLE(2)
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
40. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
EXAMPLE(2)
Histogram of bootstrap variance t(X) = V [X]
0.4
Density
0.2
0.0
0 1 2 3 4 5
Bootstrap estimation of variance
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
41. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
EXAMPLE(2)
Histogram of bootstrap variance t(X) = V [X]
B = 10000
n◦ of bootstrap samples
0.4
Density
0.2
0.0
0 1 2 3 4 5
Bootstrap estimation of variance
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
42. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
EXAMPLE(2)
Histogram of bootstrap variance t(X) = V [X]
B = 10000
n◦ of bootstrap samples
0.4
Density
mean = 2.191
0.2
0.0
0 1 2 3 4 5
Bootstrap estimation of variance
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
43. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
EXAMPLE(2)
Histogram of bootstrap variance t(X) = V [X]
B = 10000
n◦ of bootstrap samples
0.4
Density
mean = 2.191
se = 0.649
0.2
0.0
0 1 2 3 4 5
Bootstrap estimation of variance
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
44. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
R CODE
## s i m u l a t i o n poisson data
s e t . seed ( 5 9 2 )
x= r p o i s ( 1 5 , lambda =2)
B=10000
## c r e a t e t h e b o o t s t r a p f u n c t i o n
b o o t s t r a p <− f u n c t i o n ( data , nboot , t h e t a , . . . )
{
z <− l i s t ( )
datab <−
m a t r i x ( sample ( data , s i z e = l e n g t h ( data ) ∗nboot , r e p l a c e =TRUE) , nrow=nboot )
e s t b <− a p p l y ( datab , 1 , t h e t a , . . . )
e s t <− t h e t a ( data , . . . )
z$ e s t <− e s t
z$ d i s t n <− e s t b
z$ b i a s <− mean ( e s t b)−e s t
z$se <− sd ( e s t b )
z
}
## E s t i m a t i n g t h e mean
X1= b o o t s t r a p ( x , B , t h e t a =mean )
h i s t ( X1$ d i s t n , main= " Histogram o f b o o t s t r a p mean " , prob=T ,
x l a b = " B o o t s t r a p e s t i m a t i o n o f mean " )
mean ( X1$ d i s t n )
X1$se
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
45. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
Outline
1 INTRODUCTION
2 DESCRIPTION OF METHODS
METHOD 1
METHOD 2
METHOD 3
3 BOOTSTRAP IN REGRESSION MODELS
4 BAYESIAN BOOTSTRAP
5 DISCUSSION
6 BAG OF LITTLE BOOTSTRAP
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
46. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
METHOD 3 - RELATIONSHIP WITH THE JACKKNIFE
Pi∗ = Ni∗ /n P∗ = (P1 , P2 , . . . , Pn )
∗ ∗ ∗
E∗ P∗ = e/n Cov∗ P∗ = I/n2 − e e/n3
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
47. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
USING TAYLOR EXPANSION
ˆ
R(P∗ ) = R(X∗ , F ) evaluate in P∗ = e/n
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
48. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
USING TAYLOR EXPANSION
ˆ
R(P∗ ) = R(X∗ , F ) evaluate in P∗ = e/n
1
R(P∗ ) = R(e/n) + (P∗ − e/n)U + (P∗ − e/n)V(P∗ − e/n)
2
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
49. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
USING TAYLOR EXPANSION
ˆ
R(P∗ ) = R(X∗ , F ) evaluate in P∗ = e/n
1
R(P∗ ) = R(e/n) + (P∗ − e/n)U + (P∗ − e/n)V(P∗ − e/n)
2
. . .
.
.
. . .
. . . .
∂R(P∗ ) . .
∂ 2 R(P∗ )
U = ∂P ∗ V = .
. .
.
i ∂Pi∗ ∂Pj∗
.
. .
. .
. .
.
. P∗ =e/n . . . P∗ =e/n
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
50. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
DERIVATION OF BOOTSTRAP EXPECTATION AND
VARIANCE
P∗
R(P∗ ) = R n ∗
i=1 Pi
eU = 0 eV = −nU eVe = 0
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
51. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
DERIVATION OF BOOTSTRAP EXPECTATION AND
VARIANCE
P∗
R(P∗ ) = R n ∗
i=1 Pi
eU = 0 eV = −nU eVe = 0
1 1 ¯
E∗ R(P∗ ) = R(e/n) + tr V I/n2 − e e/n3 = R(e/n) + V
2 2n
n
Var∗ R(P∗ ) = U I/n2 − e e/n3 U = Ui2 /n2
i=1
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
52. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
RESULTS
ˆ
BiasF θ(F ) ≈ 1 ¯
2n V
ˆ n 2 2
VarF θ(F ) ≈ i=1 Ui /n
The results agree with those given by Jaeckel’s infinitesimal
jackknife
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
53. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
Outline
1 INTRODUCTION
2 DESCRIPTION OF METHODS
METHOD 1
METHOD 2
METHOD 3
3 BOOTSTRAP IN REGRESSION MODELS
4 BAYESIAN BOOTSTRAP
5 DISCUSSION
6 BAG OF LITTLE BOOTSTRAP
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
54. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
REGRESSION MODELS
Xi = gi (β) + i i ∼F i = 1, . . . , n
Having observed X = x we compute the estimate of β
n
2
ˆ
β = minβ ˆ
xi − gi β
i=1
ˆ 1 ˆ
F : mass at ˆi = xi − gi β
n
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
55. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
BOOTSTRAP SAMPLE
Xi∗ = gi β +
ˆ ∗ ∗ ˆ
∼F
i i
n
2
ˆ
β ∗ : minβ xi∗ − gi β
ˆ
i=1
β ∗1 , β ∗2 , β ∗3 , . . . , β ∗B
ˆ ˆ ˆ ˆ
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
56. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
LINEAR MODEL
gi (β) = ci β CC=G
β = G−1 C X has mean β and covariance matrix σF G−1
ˆ 2
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
57. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
LINEAR MODEL
gi (β) = ci β CC=G
β = G−1 C X has mean β and covariance matrix σF G−1
ˆ 2
ˆ
β ∗ = G−1 C X∗ has boostrap mean and variance
E∗ β ∗ = β
ˆ ˆ Cov∗ β ∗ = σ 2 G−1
ˆ ˆ
2
n ˆ
where σ 2 =
ˆ i=1 xi − g β /n
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
58. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
JACKKNIFE IN LINEAR REGRESSION
Applying the infinitesimal jackknife in a linear regression model,
Hinkley derive the approximation of
n
Cov β ≈ G−1
ˆ ci ci ˆ2 G−1
i
i=1
Jackknife methods ignore that the errors i are assumed to
have the same distribution for every value of i
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
59. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
Outline
1 INTRODUCTION
2 DESCRIPTION OF METHODS
METHOD 1
METHOD 2
METHOD 3
3 BOOTSTRAP IN REGRESSION MODELS
4 BAYESIAN BOOTSTRAP
5 DISCUSSION
6 BAG OF LITTLE BOOTSTRAP
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
60. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
DEFINITION OF BAYESIAN BOOTSTRAP (D. Rubin
1981)
Bayesian Bootstrap
In bootstrap we consider sample cdf is population cdf
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
61. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
DEFINITION OF BAYESIAN BOOTSTRAP (D. Rubin
1981)
Bayesian Bootstrap
In bootstrap we consider sample cdf is population cdf
Each BB replications generates a posterior probability for
each xi
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
62. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
DEFINITION OF BAYESIAN BOOTSTRAP (D. Rubin
1981)
Bayesian Bootstrap
In bootstrap we consider sample cdf is population cdf
Each BB replications generates a posterior probability for
each xi
1
The posterior probability of each xi is centered at n but has
variability
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
63. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
BB REPLICATION
BB replication
(n − 1) Unif (0, 1) u(0) = 0 e u(n) = 1
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
64. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
BB REPLICATION
BB replication
(n − 1) Unif (0, 1) u(0) = 0 e u(n) = 1
gl = u(l) − u(l−1)
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
65. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
BB REPLICATION
BB replication
(n − 1) Unif (0, 1) u(0) = 0 e u(n) = 1
gl = u(l) − u(l−1)
Attach the vector (g1 , . . . , gn ) to the data X
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
66. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
CONCEPTUAL DIFFERENCE
Bayesian Bootstrap
Simulates the posterior distribution of the parameter
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
67. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
CONCEPTUAL DIFFERENCE
Bayesian Bootstrap
Simulates the posterior distribution of the parameter
Classical Bootstrap
Simulates the estimated sampling distribution of a statistic
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
68. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
BB EXAMPLE
Dichotomous Example
The parameter is θ = Pr {Xi = 1} and let n1 number of Xi = 1
Call P1 the sum of the n1 probabilities assigned to the xi = 1
(g1 , . . . , gn ) ∼ Dirichlet(1, . . . , 1) ⇒ P1 ∼ Beta(n1 , n − n1 )
Note: Beta(n1 , n − n1 ) is the posterior distribution when the
prior is P(θ) ∝ [θ(1 − θ)]−1
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
69. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
Outline
1 INTRODUCTION
2 DESCRIPTION OF METHODS
METHOD 1
METHOD 2
METHOD 3
3 BOOTSTRAP IN REGRESSION MODELS
4 BAYESIAN BOOTSTRAP
5 DISCUSSION
6 BAG OF LITTLE BOOTSTRAP
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
70. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
INFERENCES PROBLEMS
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
71. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
INFERENCES PROBLEMS
Is it possible that all the values of X have been observed?
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
72. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
INFERENCES PROBLEMS
Is it possible that all the values of X have been observed?
Is it reasonable to assume a priori independent
parameters, constrained only to sum to 1, for these values?
Using the gap to simulate the posterior distributions of
parameters may no longer work
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
73. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
INFERENCES PROBLEMS
Is it possible that all the values of X have been observed?
Is it reasonable to assume a priori independent
parameters, constrained only to sum to 1, for these values?
Using the gap to simulate the posterior distributions of
parameters may no longer work
so..
BB and bootstrap cannot avoid the sensitivity of inference to
model assumptions
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
74. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
CONCLUSION
Knowledge of the context of a data set may make the
incorporation of reasonable model constraints obvious and
bootstrap may be useful in particular contexts
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
75. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
CONCLUSION
Knowledge of the context of a data set may make the
incorporation of reasonable model constraints obvious and
bootstrap may be useful in particular contexts
In general
"There are no general data analytic panaceas that
allow us to pull ourselves up by our bootstraps"
Donald Rubin
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
76. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
Outline
1 INTRODUCTION
2 DESCRIPTION OF METHODS
METHOD 1
METHOD 2
METHOD 3
3 BOOTSTRAP IN REGRESSION MODELS
4 BAYESIAN BOOTSTRAP
5 DISCUSSION
6 BAG OF LITTLE BOOTSTRAP
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
77. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
BLB (M. Jordan 2012)
When n gets large computational cost is large
Expected numbers of distinct points in a resample is ∼ 0.632n
BLB Procedure
Divide the dataset in s subset of dimension b, with b < n
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
78. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
BLB (M. Jordan 2012)
When n gets large computational cost is large
Expected numbers of distinct points in a resample is ∼ 0.632n
BLB Procedure
Divide the dataset in s subset of dimension b, with b < n
From each subset we draw r samples with replacement of
dimension n
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
79. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
BLB (M. Jordan 2012)
When n gets large computational cost is large
Expected numbers of distinct points in a resample is ∼ 0.632n
BLB Procedure
Divide the dataset in s subset of dimension b, with b < n
From each subset we draw r samples with replacement of
dimension n
Compute for each subset the estimator quality assessment
(e.g the bias) indicated with ξ
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
80. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
BLB IMAGE
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
81. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
FINALLY...
if we choose b = n0.6 ad we have a dataset of 1TB, the
subsamples contains at most 3981 distinct points and have size
at most 4GB
Like the bootstrap
Share bootstrap’s consistency
Automatic : without knowledge of the internals θ
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
82. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
FINALLY...
if we choose b = n0.6 ad we have a dataset of 1TB, the
subsamples contains at most 3981 distinct points and have size
at most 4GB
Like the bootstrap
Share bootstrap’s consistency
Automatic : without knowledge of the internals θ
Beyond the bootstrap
Can explicity control b
Generally faster than the bootstrap and requires less total
computation
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
83. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
References I
B. Efron.
Bootstrap Methods: Another Look at the Jackknife.
The Annals of Statistics, Vol. 7, No. 1, (Jan. 1979), pp. 1-26.
D.B. Rubin.
The Bayesian Bootstrap.
The Annals of Statistics, Vol. 9, No.1, pp. 130-134.
M. Jordan.
The Big Data Bootstrap.
Proceedings of the 29th International Conference on
Machine Learning (ICML).
Marco Brandi Bootstrap Methods: Another Look at the Jackknife
84. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
THANK YOU
FOR
YOUR ATTENTION
Marco Brandi Bootstrap Methods: Another Look at the Jackknife