This four-day course gives a solid practical and intuitive understanding of the fundamental concepts of discrete and continuous probability. It emphasizes visual aspects by using many graphical tools such as Venn diagrams, descriptive tables, trees, and a unique 3-dimensional plot to illustrate the behavior of probability densities under coordinate transformations. Many relevant engineering applications are used to crystallize crucial probability concepts that commonly arise in aerospace CONOPS and tradeoffs

1. Course Sampler From ATI Professional Development Short Course
Fundamentals of Engineering Probability
Visualization Techniques & MatLab Case Studies
Instructor:
Dr. Ralph E. Morganstern
ATI Course Schedule: http://www.ATIcourses.com/schedule.htm
ATI's Engineering Probability: http://www.aticourses.com/Fundamentals_of_Engineering_Probability.htm

2. www.ATIcourses.com
Boost Your Skills 349 Berkshire Drive
Riva, Maryland 21140
with On-Site Courses Telephone 1-888-501-2100 / (410) 965-8805
Tailored to Your Needs
Fax (410) 956-5785
Email: ATI@ATIcourses.com
The Applied Technology Institute specializes in training programs for technical professionals. Our courses keep you
current in the state-of-the-art technology that is essential to keep your company on the cutting edge in today’s highly
competitive marketplace. Since 1984, ATI has earned the trust of training departments nationwide, and has presented
on-site training at the major Navy, Air Force and NASA centers, and for a large number of contractors. Our training
increases effectiveness and productivity. Learn from the proven best.
For a Free On-Site Quote Visit Us At: http://www.ATIcourses.com/free_onsite_quote.asp
For Our Current Public Course Schedule Go To: http://www.ATIcourses.com/schedule.htm

3. Fundamental Probability Concepts
• Probabilistic Interpretation of Random Experiments (P)
– Outcomes: sample space
– Events: collection of outcomes (set theoretic)
– Probability Measure: assign number “probability” P ε [0,1] to event
• Dfn#1-Sample Space (S): Fine-grained enumeration (atomic - parameters)
– List all possible outcomes of a random experiment
– ME - Mutually exclusive - Disjoint “atomic”
– CE - Collectively exhaustive - Covers all outcomes
• Dfn#2- Event Space (E): Coarse-grained enumeration (re-group into sets)
– ME & CE List of Events
S (all outcomes)
Atomic Outcomes
Events: A,B,C ME but not CE A D
(Disjoint by dfn)
Events: A,B,C ,D both ME & CE
C
B
14 INDEX
Discrete parameters uniquely define the coordinates of the Sample Space (S) and the collection of all
parameter coordinate values defines all the atomic outcomes. As such atomic outcomes are mutually
exclusive (ME) and collectively exhaustive (CE) and constitute a fundamental representation of the Sample
Space S.
By taking ranges of the parameters such as A, B, C, and D, one can define a more useful Event Space which
should consist of ME and CE events which cover all outcomes in S without overlap as shown in the figure.
14

4. Fair Dice Event Space Representations
d2
• Coordinate Representation: 6
– Pair 6-sided dice 5
A: d1=3, d2 =arb.
4
– S={(d1,d2): d1,d2 = 1,2,…,6} 3
2 C: d1=d2
– 36 Outcomes Ordered pairs 1
d1
1 2 3 4 5 6
B: d1+d =7
• Matrix Representation: 1  [1 2 3 4 5 6]  (1,1) (1,2) (1,3) (1,4) (1,5) (1,2 ) 
6
  (2,1) (2,2) (2,3) (2,4) (2,5) (2,6)
– Cartesian Product: 2  
3
 
= (3,1)

(3,2) (3,3) (3,4) (3,5) (3,6)

– {d1} x {d2} = d1 d2T 4 (4,1) (4,2) (4,3) (4,4) (4,5) (4,6)
  (5,1) (5,2) (5,3) (5,4) (5,5) (5,6)
5  
  (6,1)
 (6,2) (6,3) (6,4) (6,5) (6,6) 
6
• Tree Representation: d2
d1 (1,1)
(1,2)
1 (1,3) 36 Outcomes
(1,4) Ordered Pairs
2 (1,5)
3 (1,6)
• Polynomial Generator for Sum Start
4
2 Dice 5 (6,1)
(6,2)
( x1 + x 2 + x3 + x 4 + x5 + x 6 ) 2 = 1x 2 + 2 x3 + 3 x 4 + 4 x5 + 5 x 6 + 6 x 7 6 (6,3)
(6,4)
Exponents represent + 5 x8 + 4 x9 + 3 x10 + 2 x11 + 1x12 (6,5)
(6,6)
6-sided die face numbers Exponents represent pair sums
Coefficients represent #ways 16
It is helpful to have simple visual representations of Sample and Event Spaces
For a pair of 6-sided dice, coordinate, matrix, and tree representations are all useful representations. Also
the polynomial generator for the sum of a pair of 6-sided dice immediately gives probabilities for each sum.
Squaring the polynomial (x1+x2+x3+x4+x5 +x6)2 yields a generator polynomial whose exponents represent
all possible sums for a pair of 6-sided dice S={2,3,4,5,6,7,8,9,10,11,12}and whose coefficients C=
{1,2,3,4,5,6,5,4,3,2,1} represent the number of ways each sum can occur. Dividing by the coefficients C by
the total #outcomes 62 = 36 yields the probability “distribution” for the pair of dice.
Venn diagrams for two or three events are useful; for example, the coordinate representation in the top
figure can be used to visualize the following events
A: {d1 = 3 and d2 = arbitrary, B= {d1 + d2 = 7}, and C= {d1 = d2}
Once we display these two events on the coordinate diagram their intersection properties are obvious, viz.,
both A & B and A & C intersect, albeit at different points, while B & C do not intersect (no point
corresponding to sum=7 and equal dice values). More than three intersecting sets, become problematic for
Venn diagrams as the advantage of visualization is muddled somewhat by the increasing number of
overlapping regions in theses cases (see next two slides).
16

5. Venn Diagram for 4 Sets
4C = (4C1 4-Singles) – (4C2 6-Pairs) + (4C3 4-Triples ) - ( 4C4 1-Quadruple)
0
A B
AB
BD AC
ABD ABC
AD ABCD BC
ACD BCD
CD
D C
17
As we go to Venn diagrams with more than 3 sets the labeling of regions becomes a practical limitation to
their use. In this case of 4 sets A,B,C, D, the labeling is still pretty straightforward and usable.
The 4 singles A,B,C,D are labeled in an obvious manner at the edge of each circle.
The 6 pairs AB,AC,AD,BC,BD,CD are labeled at the intersection of two circles. The 4 triples ABC, ABD,
BCD, ACD are labeled within “curved triangular areas” corresponding to the intersections of three circles.
The 1 quadruple ABCD is labeled within the unique “curved quadrilateral area” corresponding to the
intersection of all four circles.
17

6. Trivial Computation of Probabilities of Events
sum = d1 + d2
d2
Ex#1 Pair of Dice E1
S={(d1,d2): d1,d2 = 1,2,…,6} 6
12
5 11 E2
10
E1={(d1,d2): d1+d2 ¥ 10} 4 9
8
P(E1)=6/36=1/6 3 7
6
2 5
E2={(d1,d2): d1+ d2 = 7} 4
P(E2)=6/36=1/6 1 3
2 d1
1 2 3 4 5 6
Ex#2 Two Spins on Calibrated Wheel
S={(s1,s2): s1,s2 ε [0,1]} s2
E1={(s1,s2): s1+s2 ¥ 1.5}--> P(E1) = ----- =.52/2=1/8 1
1
E1
0.5 E3
E2={(s1,s2): s2 § .25} --> P(E2)=1(.25)/1=.25
E2
0 s1
E3={(s1,s2): s1= .85; s2= .35}--> P(E3)=0/1=0 0 0.5 1
20
For equally likely atomic events the probability of any outcome Event is easily computed as the (#atomic
outcomes in Event)/(total # outcomes). For a pair of dice, the total # of outcomes is 6*6=36 and hence
simple counting of the # points in E /36 yields P(E), etc.
Two spins on a calibrated wheel [0, 1) can be represented by the unit square in the (s1 , s2)-plane and an
analogous calculation can be performed to obtain the probability for the event E by dividing the area
covered by the event by the area of the event space (“1”): P(E)= area(E)/ 1.
20

7. DeMorgans’ Formulas - Finite Unions and Intersections
i) Compl(Union) = Intersec(Compls): ( E1 ∪ E2 ∪
c
∪ En ) c = E1 ∩ E2 ∩
c
∩ En
c
c c c
ii) Compl(Intersec) = Union(Compls): ( E1 ∩ E2 ∩ ∩ En ) c = E1 ∪ E2 ∪ ∪ En
Useful Forms:
A∪ B
i’) Union expressed ( A ∪ B) c = Ac B c
Visualization
Compl(Union) Intersec(Compl) ( A ∪ B)c
as an Intersection
(( A ∪ B) )
c c
= A ∪ B = ( Ac B c ) c A Ac Intersect
grey areas
B Bc Ac & B c
ii’) Intersection ( AB) c = Ac ∪ B c Ac B c Yields one
Union(Compl) grey area Ac B c
expressed as a Union Compl(Intersec)
with A and
B excluded
(( AB) )
c c
(
= AB = Ac ∪ B c )c
Taking its
complement ( Ac B c )c
yields white
area, i.e., A ∪ B
24 INDEX
DeMorgan’s Laws for the complement of finite unions and intersections states that
i) The complement of unions equals the intersections of the complements, and
ii) The complement of intersections equals the union of complements
The alternate forms obtained by taking the complements of the original equations are often more useful
because they give a direct decomposition of the union and the intersection of two or more sets
i’) The union equals the complement of the (intersection of complements)
ii’) The intersection equals the complement of the (union of complements)
A graphical construction of A U B = (Ac Bc)c is also shown in the figure..
Ac and Bc are the two shaded areas in the middle planes which exclude A and B respectively (white) ovals
Intersecting these two shaded areas and taking the complement leaves the white oval areas which is A U B
24

8. Set Algebra Summary Graphic
Union A ∪ B = A ∪ Ac B
= B ∪ Bc A Union AUB
“A-B” “B-A”
Intersection A ∩ B = A ⋅ B = AB A Bc A
AB B
Ac B
x ∈ AB iff x ∈ A & x ∈ B
Intersection
Difference A − B ≡ A ∩ B c = AB c
x ∈ A − B iff x ∈ A and x ∉ B Differences
DeMorgans A ∪ B = ( Ac B c )c ( A ∪ B )c = Ac B c
means
( )
c
AB = Ac ∪ B c complement of (At least one) = (not any)
27
This summary graphic illustrates the set algebra for two sets A , B and their union intersection and
difference.
DeMorgans Law can be interpreted as saying “the complement of (“at least one”) is “not any”
Associativity and commutivity of the two operations allows extension to more than two sets.
27

9. Basic Counting Principles
Principle #0: Take Case n=3-4; generalize to n Binomial Expansion: (a+b)3 (a+b)n
Repetitions Allowed
Principle #1: Product Rule for Sub-experiments: 6- Bins
= 263 ⋅103
m Num Suit
Licenses
⋅ nm = ∏ nk
26 26 26 10 10 10
n = n1 ⋅ n2
H
1 D
S
C
H
5
16- Bins
k =1 Start 2 D
Binary
S
2
216 = 65,536
C
H
13 D
2 2 2 2 ... 2
Generate “tree” of outcomes
S
C Digits
#ways: 13 * 4 = 52
No Repetitions
Principle #2: Perm n distinguish-obj take k k=n Arrange 11 Travel 5 Cooking 4 Garden
All Books
n! “Fill k-bins” 11! 5! 4!
n
Pk = (n) k = 3! Permute Groups
(n − k )! k<n 11 Travel Books
in 5 bins 11| 10| 9 |8 |7
Principle #3:Perm n-obj take n with r - Arrange 4!
groups of indistinguishable objects Letters
“TOOL” = 12
2!⋅1!⋅1!
hable
 # Distinguis n! 10!
 Sequences  = n !⋅n !⋅ ⋅ n !
  r − groups {4”r”, 3”s”, 2”o”, 1 “t”}
4!⋅3!⋅2!⋅1!
= 12,600
  1 2 r
Principle #4: Combination of n-objects take k Committee of 4 22! 22!
C4 =
22
= = 7315
from 22 people (22 − 4)!4! 18!4!
n n!
n
Ck =   =
 k  k ! ( n − k )! k ≤ n Order not
  Committee of 3 {2M, 1F} 6⋅5
important!
from {6M, 3F}
6
C2 ⋅3 C1 = ⋅ 3 = 45
2!
= Principle #3 with {taken , not taken} not counted 28 INDEX
Outcomes must be distinguished by labels. They are characterized by either i) distinct orderings or ii)
distinct groupings. A grouping consists of objects with distinct labels; changing order within a group is not
a new group, but is a new permutation. The four basic counting principles for groups of distinguishable
objects are summarized and examples of each are displayed in the table.
Principle#0: This is practical advice to solve a problem with n= 2,3,4 objects first and then generalize the
“solution pattern” to general n.
Principle#1: This product rule is best understood in terms of the multiplicative nature of outcomes as we
“branch out” on a tree. For a a single draw from a deck of cards there are 13 “number” branches and, in
turn, each of these has 4 “suit” branches yielding 13*4 =52 distinguishable cards or outcomes.
Principle#2: Permutation (ordering) of n objects take k at a time is best understood by setting up “k-
containers” putting one of “n” in the first, one of “n-1” , ... and finally one of “n-k+1” in the kth container.
The total #ways is obtained by the product rule as n*(n-1)*...*(n-k+1) = n!/(n-k)!
Principle#3: Permutation of all ”n” objects consisting of “r “ groups of indistinguishable objects {3 t , 4
s 5 u}. If all objects were distinguishable then the result would be n! permutations; however permutations
within the r groups does not create new outcomes and therefore we divide by factorials of the numbers in
each group to obtain n!/(n1! n2! ... nr!)
Principle#4: Combination of n objects take k is related to Principles#2, #3. There are n! permutations;
ignoring permutations within r= 2 groups {“taken” , “not taken”} yields n!/(n! (n-k)!)
28

10. Counting with Replacement
Refills Drop Down
Select “B” from Alphabet and Replace A
A
B
B ... Y
Y
Z
Z
Always have 26 letters to choose from A
A
B
B
Y
Y
Z
Z
23 =8 distinct 4 distinct
Permutation of “n” obj with (# drws)
orderings groupings
replacement taken “k” at a time
n
Pk = # replaceable
objects = nk A {AAA} 3 “A”
B {AAB} 2 “A”& 1”B”
A
n n n n n…n A {ABA} 2 “A”& 1”B”
A B B {ABB} 2 “B”& 1”A”
Bin# 1 2 3 …k S A A {BAA} 2 “A”& 1”B”
B
n=2 , k=3 B B {BAB} 2 “B”& 1”A”
A {BBA} 2 “B”& 1”A”
B {BBB} 3 “B”
Combination of “n” obj with
replacement taken “k” at a time
effective # objects
 n + k − 1  n + k − 1
n
Ck =
/ n + (k-1)
= n + k −1
Ck =  = 
Note: “k” can be larger than “n”
(draw k)  k   n −1 
Example: From 2 objects {A, B} choose 3 with replacement (Only Way!)
After each draw of an A or B “drop 4 Outcomes
down a replacement” add 1 after each A B A/B A/B
{AAA},{BBB}
draw except last 4! {ABB},{AAB}
(effective # objects) = 2 +(3-1)=4
2
C3 = 2+3−1C3 = 4C3 =
/ =4
3! 1!
41 INDEX
Counting permutations and combinations with replacement is analogous to a candy machine purchase in
which a new object drops down to replace the one that has been drawn, thus giving the same number of
choices in each draw.
Permutation of n obj taken k at a time with replacement: Each of the k draws has the same number of
outcomes n because of replacement, the result is n*n*n... *n = nk and is written nPk with an “over-slash” on
the permutation symbol. The case n=2, k=3 of 3 draws with 2 replaceable objects {A,B} shows the slash-
2
P3 =23 = 8 permutations that result.
Combination of n obj taken k at a time with replacement: For n=2, k=3, 2 take 3 does not make any
sense. However, with replacement, it does since each draw except the last drops down an identical item and
hence the number of items to choose from becomes n +(k-1) and slash-nCk = n+(k-1)Ck. The tree verifies this
formula and explicitly shows that there are 4 distinct groupings {3A, 3B, 2A1B, 1A2B} exactly the number
of combinations with replacement given by the general formula slash-2C3 = 2+(3-1)C3 = 4C3 =4
41

11. II) Fundamentals of Probability
1. Axioms
2. Formulations: Classical, Frequentist, Bayesian, Ad Hoc
3. Adding Probabilities: Inclusion / Exclusion, CE & ME
4. Application of Venn Diagrams & Trees
5. Conditional Probability & Bayes’ “Inverse Probability”
6. Independent versus Disjoint Events
7. System Reliability Analysis
47
As a theory, Probability is based on a small set of axioms which set forth fundamental properties of
construction.
In practice, probability may be formulated theoretically, experimentally, or subjectively, but must always
obey the basic Axioms.
Evaluating probabilities for events, is naturally developed in terms of their unions and intersections using
Venn Diagrams, Trees and Inclusion/Exclusion techniques.
Conditional probabilities, their inverses (Bayes’ theorem), and the dependence between two or more events
flow naturally from the basic axioms of probability.
System reliability analysis utilizes all these fundamental concepts
47

12. Inclusion / Exclusion Ideas
ME Events A,B - Disjoint AB= φ A B P(A∩B) = P(A) + P(B) No intersections
”Add Prob”
No intersections
Intersect: “CE, not ME” “Recast” as Disjoint Union “CE & ME”
Not Disjoint AB∫φ
A A B-A
B ∫
AB
P(A∩B) = P(A) + P(B-A) = P(A) + P(BAc)
Intersection “AB” Counted Twice!! P(A∩B) ∫ P(A) + P(B)
B = B ⋅ S = B ⋅ ( A ∪ Ac ) = BA ∪ BAc
Subtract “P(AB)” from sum; count only once
A
BAC B
P ( A ∪ B ) = P ( A) + P ( B ) − P ( AB ) AB
P( BAc ) = P( B) − P( AB)
Generalization by Induction: let D = B ∪ C
P ( A ∪ B ∪ C ) = P ( A ∪ D ) = P ( A) + P ( D) − P ( AD ) = P ( A) + P ( B ∪ C ) − P( A ⋅ ( B ∪ C ))
= P ( A) + {P ( B ) + P (C ) − P ( BC )} − {P ( AB ) + P ( AC ) − P ( ABAC )}
Inclusion /
P ( A ∪ B ∪ C ) = P ( A) + P ( B ) + P (C ) − P ( AB ) − P ( AC ) − P ( BC ) + P ( ABC ) Exclusion
add singles subtract pairs add triples
54 INDEX
It is important to realize that although probabilities are simply numbers that add, the probability of the
union of two events P(A U B) is not equal to the sum of individual probabilities for the two events P(A) +
P(B).
This is because points in this overlap region AB are counted twice; to correct for this one needs to subtract
out “once” the double counted points in the overlap yielding P(A U B) = P(A) + P(B)-P(AB).
Only in the case of non-intersection AB = φ does the simple sum of probabilities hold.
The generalization for a union of three or more sets alternates inclusion and exclusion; for A,B,C the
probability P(AUBUC) adds the singles, subtracts the doubles and adds the triple as shown.
54

13. Venn Diagram Application: Inclusion/Exclusion
Given following information find how many club
members play at least one sport T or S or B T (36) TS (22) S (28)
Club: 36 T , 28 S, 18 B
TSB
(4)
SB (9)
Let N= Total # members (unknown) TB (12)
36 28 18 B (18)
Write Probabilities as P(T) = ; P(S) = ; P(B) = ; etc.
N N N CLUB
Method 1: Subs into Formula for Union
P ( T ∪ S ∪ B) = P (T ) + P( S ) + P( B ) − P (TS ) − P(TB ) − P ( BS ) + P (TBS )
36 28 18 22 12 9 4 TS (22)
= + + − − − + T (36) STc (6)
N N N N N N N
43 18 1
= Thus 43 of “N” Club Members play 6 TSB
N at least one sport. (N is irrelevant) (4) 5
8 SB (9)
Method 2: Disjoint Union - Graphical TB (12) 1
T ∪ S ∪ B = T ∪ ST ∪ BT Sc c c
BTcSc (1)
CLUB
P(T ∪ S ∪ B) = P(T ) + P( ST c ) + P( BT c S c )
36 6 1 43
= + + =
N N N N
68 INDEX
This example illustrates the ease by which a Venn diagram can display the probabilities associated with the
various intersections of 3 sets T, S, and B.
The number of elements in each of the 7 distinct regions is easily read off the figure; they are required to
establish the total number in their union T U S U B via the inclusion/exclusion formula.
Another method of finding P(T U S U B ) is to decompose the union T U S U B into a union of disjoint sets
T* U S* U B* for which the probability is additive, i.e., P(T* U S* U B* ) = P(T*) + P(U*) + P(B*).
68

14. Matching Problem – 1
“N” men throw hats onto floor; Each man in turn randomly draws a hat
a) No Matches - Find Probability None draw own hat.
Let Event Ei = ith man chooses his own hat ; compute: P(0 − matches) = 1 − P( E1 ∪ E2 ∪ ∪ EN )
1|2|3|… | k | k+1 | … |N Hats
i1 |i2 | i3 | … | in in+1 | in+2 | in+3 | … | iN Men
Probability that
M1 & M2 &...&Mn irrespective of what
n “Ei s” choose own hats (N-n) Does not Matter
draw own hats other men draw
(Matched or Not Matched )
Total # of“n-tuple” N # perms ( N − n)!
  P( Ei1 Ei2 Ein ) = =
selections from N n Total# perms N!
 
 N  ( N − n)! N! ( N − n)! 1
Sum Joint Probabilities
∑ P( Ei1 Ei2 Ein ) =   ⋅ =
n !( N − n)! N !
=
over all “n-tuples” n −tuples
All n-tuples Eq. Likely
n N! n!
 
P (0 − Matches ) = 1 − P ( E1 ∪ E2 ∪ E3 ) = 1 −  ∑ P ( Ei1 ) − ∑ P ( Ei1 Ei2 ) + ∑ P( Ei1 Ei2 Ei3 ) = 1 − {1 − 2! + 3!} =
1 1 1
3
1− tuples pairs triples 
P(0 − matches) = 1 − P( E1 ∪ E2 ∪ ∪ EN ) = 1
−
1
+
1
−
2! 3! 4! 5!
1
+ + ( −1)
N 1
N!
 e−1
N →∞
→
b) k- Matches
Poisson with success rate λ=1/N & “time
 k! ⋅ e−1
→1
1 1 1 N −k 1 
 − + + + ( −1) 
( N − k )! 
P(k matches) =  2! 3! 4!
N→∞ intvl” t = N samples; a=λ *t =(1/N)*N =1
k!
69 INDEX
Here is an example that requires the inclusion/exclusion expansion for a large number of intersecting sets.
Since it becomes increasingly difficult to use Venn diagrams for a large number of intersecting sets, we
must use the set theoretic expansion to compute the probability. We shall spend some time on this problem
as it is very rich in probability concepts.
The problem statement is simple enough: “N men throw their hats onto the floor; each man in turn
randomly draws a hat. “
a) What is the probability that no man draws his own hat?
b) What is the probability of exactly k-matches?
Key ideas: define Event Ei = ith man selects his own hat
then take union of N sets E1 U E2 U ... U EN and
P(no-matches)=1- P(E1 U E2 U ... U EN)
The expansion of the P(E1 U E2 U ... U EN) involves addition and subtraction of P(singles), P(pairs),
P(triples), etc. ( The events Ei are CE but not ME so you cannot simply sum up the P(Ei ) for k singles to
obtain an answer to part b)) .
This slide shows a key part of the proof which establishes the very simple result that the sum over singles,
P(singles) = 1/(1!); sum over pairs is P(pairs)= 1/(2!) ; sum over triples is P(triples)=1/(3!); sum over 4-
tuples, P(4-tuples) = 1/(4!); ... sum over N-tuples, P(N-tuple) = 1/(N!).
Limit as N large approaches a Poisson Distribution with success rate for each draw λ=1/N and data length
t =N i.e., parameter a =λ t =1
69

15. Man Hat Problem n =3 Tree/Table Counting
M#1 M#2 M#3 M.E. Match
Tree#1 Drw#1
Drw#2 Drw#3 Outcomes Outcomes
M#1 M#2 M#3 #Matches
E2 1
2 3 E3 {E1 E2 E3 } triple 1 2 3 3
1/2 Br#1
1
E1 1/2 3 2 c
{E1 E2 E3 }
c
single 1 3 2 1
1/3 1 1/2
1
1
3 E3 {E1c E2 c E3 } single 2 1 3 1
E1C Br#2
Start 1/3 2 1/2 1
1
c c
{E1 E2 E3 }
c
No-match 2 3 1 0
3
1/3 1/2
3 1
1
2 c c
{E1 E2 E3 }
c
No-match 3 1 2 0 Br#3
E1C
1/2
2 E2
1
1 c
{E1 E2 E3 }
c
single 3 2 1 1
P(Ei) = 1/3 2/6 2/6
From Table: From Tree: Connection: Matches & Events
Prob[0-matches]=2/6 Prob[0-matches]=1-Pr[E1 U E2 U E3]
Prob[1-matches]=3/6 Prob[Sgls]=P[E1]=P[E2]=P[E3]=1/3
=1-{Sum[Sngls]-Sum[Dbls]+Sum[Trpls]}
Prob[2-matches]=0/6=0 Prob[Dbls] = P[E1E2]=(1/3)(1/2)=1/6 =1-{3(1/3) -3(1/6)+1(1/6)}=2/6
Prob[3-matches]=1/6 Prob[Trpls] = P[E1E2E3]=(1/3)(1/2)=1/6
Alternate Trees Yield: P[E1E3]= P[E2E3]=1/6
75
This slide shows the complete the tree and associated table for the Man - Hat problem in which n=3 men
throw their hats in the center of a room and then randomly select a hat. The drawing order is fixed as
Man#1, Man#2, Man #3, and the 1st column of nodes labeled as circled 1, 2, 3 shows the event E1 in which
the Man#1draws his own hat, and the complementary event E1c i.e., Man#1 does not draw his own hat . The
2nd column of nodes corresponds to the remaining two hats in each branch shows the event E2 in which the
Man#2 draws his own hat; note that E2 has two contributions of 1/6 summing to 1/3. Similarly, the 3rd draw
results in the event E3 in two positions shown again summing to 1/3.
The tree yields ME & CE outcomes expressed as composite states such as {E1E2E3}, {E1E2cE3c, etc., or
equivalently in terms of the number of matches in the next column. The nodal sequence in the tree can be
translated into the table on the right which is analogous to the table we used on the previous slide. The
number of matches can be counted directly from the table as shown.
The lower half of the slide compares the “ # of matches” events with the “compound events” formed from
the “Ei”s{ no-matches, singles, pairs, and triples }. The connection between these two types of events is
based on the common event “no-matches,” i.e., the inclusion/exclusion expansion of the expression [1-
P(E1U E2U E3) ] in terms of singles doubles and triples yields P(0-matches).
75

16. Conditional Probability - Definition & Properties
ˆ
P ( AS )  2
• Definition of Conditional Probability ˆ
P( A | S ) ≡ = 
ˆ
P( S )  3
• In terms of atomic events si we can formally write
ˆ ˆ
P( ∪ si S ) ∑ P( s S )
ˆ
i
(# pts in Sˆ & A)
A = ∪ si ˆ ) = P ( A S ) = si ∈ A =
si ∈ A
=
si ∈ A
P( A | S
ˆ
P( S ) ˆ
P( S ) ˆ
P( S ) (# pts in Sˆ )
ˆ
• Note in case S = S it reduces to P(A) as it must
A B
•Asymmetry of Conditional Probability BA
P(BA)
P ( BA)  fraction  BA
P ( B | A) = = =
P ( A)  BA over A 
  A Given A
Not
Symmetrical!
P( BA)  fraction  BA
P( A | B) = = =
P( B)  BA over B 
  Given B
B
82 INDEX
The formal definition of conditional probability follows directly from the renormalization concept discussed
on the previous slide. It is simply the joint probability defined on the intersection of the set A and S-cap,
P(AS-cap) divided by the normalizing probability P(S-cap).
It can also be written explicitly in terms of a sum over atomic events given in the second equation.
Conditional probability is not symmetric because the joint probability on the intersection of A and B is
divided by probability of the conditioning set which is P(A) in one case and P(B) in the other. This is also
easily visualized using Venn diagrams where the “shape division” are obviously different in the two cases.
82

17. Examples - Coin Flips, 3-Sided Dice
nH > nT
Flip#3
Example#1: Three Coin Flips Flip#2 H {HHH}
Given the first flip is H, Find Flip#1 H
T {HHT} ˆ
S
Prob #H > #T H {HTH}
H
T T {HTT}
#H > #T
S
4 1 1 1
ˆ
P ( S ) = ; P( HHH ) = ; P ( HHT ) = ; P( HTH ) = S H H {THH}
8 8 8 8
T
T
T {THT}
3
P ( HHH ) + P ( HHT ) + P ( HTH ) 3
= 8=
H {TTH}
P (nH > nT | H ) =
ˆ)
P( S 4 4 T
8 {TTT}
Example#2: 4-Sided Dice
Given the first “die” d1= 4” d1 d2
Find Prob of Event A: “d2= 4” 1
P(d2=4| d1= 4)=? 2 S
S 3 (4,1)
ˆ 4 1 4 (4,2) ˆ
S
P ( S ) = P( d1 = 4) = ; P( 4,4) = (4,3)
16 16 d2 (4,4)
A
1 4
P(4,4) 1
P (d 2 = 4 | d1 = 4) = = 16 =
ˆ
P( S ) 4 4 3 ˆ
S Reduced
16 2 Sample space
1 d1
1 2 3 4
83
Here are two examples illustrating conditional probability.
The first involves a series of three coin flips and a tree shows all possible outcomes for the original space S.
The reduced set of outcomes conditions on the statement “ 1st draw is a head (red circle)” and S-cap only
takes the upper branch of the tree and leads to a reduced set of outcomes. The conditional probability is
computed either by considering outcomes in this conditioning space S-cap or by computing the probability
for S (the whole tree) and then renormalizing by the probability for S-cap ( upper branch).
The second example involves the throw of a pair 4-sided dice and asks for the probability that d2 =4 given
that d1=4, P(d2 =4 | d1 =4 ). The answer is obtained directly from the definition of conditional probability
and is illustrated using a tree and a coordinate representation of the dice sample space with a Venn diagram
overlay for the event (d1, d2) = (4,4) (green) and the subspace S-cap {d1=4} (red rectangle).
83

18. Probability of Winning in the “Game of Craps”
Rules for the “Game of Craps”
First Throw - dice sum=(d1+d2) Subsequent Throws - dice sum=(d1+d2)
2, 3, 12 - “Lose” (L) “Point” - “Win” (W)
7, 11 - “Win” (W) 7 “Lose” (L)
Other (O) - first time defines your “Point” = “5” say Other (O) “Throw Again”
Thr#1
2 L Thr#2 Thr#3 Thr#4
4 S=d1+d2 #Ways #Prob
3 L
36 5 2, 12 1 1/36
4 W 4
6
5 Point L 3, 11 2 2/36
36 7
36 5 W
6
26 4 P
Start
O 6 o 4, 10 3 3/36
7 W 7 L 36
36 36 5 W i
8 26 6 n 5, 9 4 4/36
O
9 7 L t
36 36 s 6, 8 5 5/36
10 26 O
11 7 6 6/36
W 36
12 L
 
4  1  2
2 3
4 4  26  4  26  4  26 
P (W | 5) = +  +   +   + =  =
36 36  36  36  36  36  36  36  1 − 26  5
 
 36 
P(W ) = P(7) + P(11) + ∑ P(W | Point )P(Point )
Points
6 2  
= + + 2  P(W | 4) P (4) + P(W | 5) P (5) + P (W | 6) P(6) = .4929
36 36  1/ 3
 3 / 36 2/5 4 / 36 5 / 11 5 / 36 

85 INDEX
Here we compute the probability of winning the game of craps previously described by the rules for the 1st
and subsequent throws given in the box and illustrated by the tree. Since there are 36 equally likely
outcomes the #ways for the two dice summing to either 2 or 12 is obviously 1/36, for 3 or 11 it is 2/36, and
the remaining sums of two dice can be read directly off the sum axis coordinate representation and are
displayed in the table on the right.
We have labeled the partial tree “given the point 5” by their conditional probabilities derived from the table.
The probability for the three outcomes W(“5”), L (“7”), “Other (not “5 or 7”) can be read off the table as
P(5)= 4/36, P(7)=6/36, P(Other)= 1-(4+6)/36 =26/36. Note that these are actually conditional probabilities;
but since the throws are independent the conditionals are the same as the a prioris as taken from the table.
The P(W|5) is obtained by summing all paths that lead to a win on this “infinite tree”. Thus the 2nd throw
yields W with probability 4/36 and the 3rd throw yields W with probability P(5|Other)P(5)=(26/36)(4/36),
and the 4th throw yields W with probability P(5|Other,Other)P(5)=(26/36)2 (4/36), ... leading to an infinite
geometric series which sums to (4/36)*1/(1-26/36)=2/5.
The total probability of winning is the sum of winning on the 1st throw (“7” or “11”) plus winning on the
subsequent throws for each possible “point.” The infinite sum for the other points is obtained in a similar
manner to that for “5” and (taking points by pairs in the table leads to the factor of two) the final result is
shown to be .4929, i.e., a 49.3% chance of winning!
85

19. Visualization of Joint, Conditional, & Total Probability
Binary Comm Signal - 2 Levels {0,1}
Binary Decision - {R0, R1}={(“0” rcvd , “1” rcvd} x = 0,1
Joint Probability
(Symmetric) 0 1 sent
P(0,R0) = P(R0,0) ovly
R1
“0” sent & R0 (“0” rcvd ) & y =R0 ,R1 R0 rcvd
R0 (“0” rcvd ) “0” sent
Conditional Probability
0R1
(Non-Symmetric) R0 ,R1 1R1
Joint
P(0|R0) ∫ P(R0|0) 0R0
1R0
“0” sent given R0 (“0” rcvd ) x = 0 ,1
P(0) = P(0, R0 ) + P(0, R1 ) P(R0 ) = P(R0 ,0) + P(R0 ,1)
R0 (“0” rcvd ) given “0” sent
Total Probability P(0) Total Probability P(R0)
sum up joint on R0,R1 sum across joint on 0,1
Conditional Probability P( R0 ,0) P( R0 ,0)
P ( R0 | 0) ≡ =
Requires Total Probability P ( 0) P( R0 ,0) + P( R0 ,1) Re-normalize
Joint Probability
P(0), P(R0), etc. P( R0 ,0) P ( R0 ,0)
P (0 | R0 ) ≡ =
P ( R0 ) P ( R0 ,0) + P ( R0 ,1)
88 INDEX
Another way to visualize the communication channel is in terms of an overlay of a Signal Plane divided
(equally) into “0”s and “1”s and a Detection Plane which characterizes how the “0”s and “1”s are detected
and is structured as shown so that when we overlay the two planes we obtain an Outcome Plane with four
distinct regions whose areas represent probabilities of the four product (joint) states { 0R0, 0R1, 1R0, 1R1}
(similar to the tree outputs).
In this representation the total probability of a “0” P(0) can be thought of as decomposed into two parts
summed vertically over the “0”-half of the bottom plane shown by the break arrow P(0) = P(0,R0) + P(0,R1)
[Note: summing on the “1”-half of the bottom plane yields P(1) = P(1,R0) + P(1,R1).]
Similarly the total probability P(R0) can be thought of as decomposed into two parts summed horizontally
over the “R0”-portion of the bottom plane shown by the break arrow P(R0) = P(R0,0) + P(R0,1); similarly
we have P(R1) = P(R1,0) + P(R1,1).
The Total Probability of a given state is obtained by performing such sums over all joint states.
88

20. Log-Odds Ratio - Add & Subtract Measurement Information
Note:
Revisit Binary Comm Channel P( R0 | 0) = .95 P ( R1 | 1) = .90 P(0)=.5
E = “1”
P( R1 | 0) = .05 P ( R0 | 1) = .10 P(1)=.5
Ec = “0”
Relation between  P (1 | R1 )  P (1 | R1 ) e L1
L1 ≡ ln 1 − P(1 | R )  ⇒ e = 1 − P(1 | R ) ⇒

L1
P(1 | R1 ) =
L1 and P(1|R1)  1  1 1 + e L1
 P(1 | R1 )   P (1)   P ( R1 | 1)   P(1)   P( R1 | 1) 
L1 ≡ ln 1 − P(1 | R )  = ln 1 − P(1)  + ln 1 − P( R | 1)  = ln P(0)  + ln P ( R | 0) 
        
 1     1     1 
≡ L0 ≡ ∆L1
 P( R1 | 1) 
Additive Meas Updates for L Lnew = Lold + ∆LR1  P (1) 
 P(0)  ; ∆LR1 = ln P( R | 0) 
Lold = ln   
   1 
Updates
Meas#1: R1 Meas#2: R0 Alternate Meas#2: R1
 .5   P( R0 | 1)   .10   P( R1 |1) 
Lold = ln  = 0 ∆LR0 = ln  .90 
 .5   P( R | 0)  = ln .95 
  
∆LR1 = ln   = ln  
 0   P( R1 | 0)   .05 
 .9  = −2.25129
∆LR1 = ln  = +2.8903
 .05  Lnew = Lold + ∆LR0 Lnew = Lold + ∆LR1
= 2.8903
= 2.8903 + (−2.25129) = .63901 = 2.8903 + 2.8903 = 5.7806
Lnew = 0 + 2.8903
e 2.8903 e.63901 e 5.7806
P(1 | R1 ) = = .947 P(1 | R1 R0 ) = = .655 P (1 | R1 R0 ) = = .997
1 + e 2.8903 1 + e.63901 1 + e 5.7806
96 INDEX
Revisiting the binary communication channel we now compute updates using the log odds ratio which are
additive updates. The update equation simply starts from the initial log odds ratio which is
Lold=ln[P(1)/P(1c)] =ln(.5/.5)=0 for the communication channel. There are two measurement types R1 and
R0 and each adds an increment ∆L determined by its measurement statistics, viz.,
R1: ∆LR1 =ln[(P(R1|1)/P(R1|1c)]=ln(.90/.05) = +2.8903 (positive “confirming”)
R0: ∆LR0 = ln[(P(R0|1)/P(R0|1c)]=ln(.10/.95)= -2.25129. (negative “refuting”)
The table illustrates how easy it is to accumulate the results of two measurements R1 followed by R0 by just
adding the two ∆Ls to obtain
Lnew= 0+2.8903-2.25129=.63901,
or alternately R1 followed by R1 to obtain
Lnew=0+2.8903+2.8903=5.7806.
These log odds ratios are converted to actual probabilities by computing P= eLnew / (1+ eLnew ) yielding .655
and .997 for the above two cases.
If we want to find the number of R1 measurements needed to give .99999 probability of “1” we need only
convert .99999 to an L =ln[(.99999)/(1-.99999)] =11.51 and divide the result by 2.8903 to find 3.98 so that
4 R1 measurements are sufficient.
96

21. Discrete Random Variables (RV) –Key Concepts
• Discrete RVs: A series of measurements of random events
• Characteristics: “Moments:” Mean and Std Deviation
• Prob Mass Fcn: (PMF), Joint, Marginal, Conditional PMFs
• Cumulative Distr Fcn: (CDF) i) Btwn 0 and 1, ii) Non-decreasing
• Independence of two RVs
• Transformations - Derived RVs
• Expected Values (for given PMF)
• Relationships Btwn two RVs: Correlations
• Common PMFs Table
• Applications of Common PMFs
• Sums & Convolution: Polynomial Multiplication
• Generating Function: Concept & Examples
122 INDEX
This slide gives a glossary of some of the key concepts involving random variables (RVs) which we shall
discuss in detail in this section. Physical phenomena are always subject to some random components so
that RVs must appear in any realistic model and hence their statistical properties provide a framework for
analysis of multiple experiments using the same model. These concepts provide the rich environment that
allows analysis of complex random systems with several RVs by defining the distributions associated with
their sums and transformations of these distributions inherent in the mathematical equations that are used to
model the system.
At any instant, a RV takes on a single random value and represents one sample from the underlying RV
distribution defined by its probability mass function (PMF). Often we need to know the probability for some
range of values of a RV and this is found by summing the individual probability values of the PMF; thus a
cumulative distribution function (CDF) is defined to handle such sums. The CDF formally characterizes the
discrete RV in terms of a quasi-continuous function that ranges between [0,1] and which has a unique
inverse.
Distributions can also be characterized by single numbers rather than PMFs or CDFs and this leads to
concepts of mean values, standard deviations, correlations between pairs of RVs and expected values.
There are a number of fundamental PMFs used to describe physical phenomena and these common PMFs
will be compared and illustrated through examples. Finally, the relationship between the sum of two RVs
and the concept of convolution and the generating function for RVs will be discussed.
122