Lecture from BMVA summer school 2014

1. Graphical
Models
for
Chains,
Trees,
and
Grids
Gabriel
Brostow
UCL

2. Sources
• Book
and
slides
by
Simon
Prince:
“Computer
vision:
models,
learning
and
inference”
(June
2012)
• See
more
on
www.computervisionmodels.com
2

3. Part
1:
Graphical
Models
for
Chains
and
Trees
3

4. Part
1
Structure
• Chain
and
tree
models
• MAP
inference
in
chain
models
• MAP
inference
in
tree
models
• Maximum
marginals
in
chain
models
• Maximum
marginals
in
tree
models
• Models
with
loops
• ApplicaOons
4
4
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Extra
Extra

5. Example
Problem:
Pictorial
Structures
5
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince

6. Chain
and
tree
models
• Given
a
set
of
measurements
and
world
states
,
infer
the
world
states
from
the
measurements.
• Problem:
if
N
is
large,
then
the
model
relaOng
the
two
will
have
a
very
large
number
of
parameters.
• SoluOon:
build
sparse
models
where
we
only
describe
subsets
of
the
relaOons
between
variables.
6
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince

7. Chain
and
tree
models
7
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Chain
model:
only
model
connecOons
between
a
world
variable
and
its
1
preceeding
and
1
subsequent
variables
Tree
model:
connecOons
between
world
variables
are
organized
as
a
tree
(no
loops).
Disregard
direcOonality
of
connecOons
for
directed
model

8. AssumpOons
8
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
We’ll
assume
that
– World
states
are
discrete
– Observed
data
variables
for
each
world
state
– The
nth
data
variable
is
condi&onally
independent
of
all
other
data
variables
and
world
states,
given
associated
world
state

9. See
also:
Thad
Starner’s
work
Gesture
Tracking
9
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince

10. Directed
model
for
chains
(Hidden
Markov
model)
10
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
CompaObility
of
measurement
and
world
state
CompaObility
of
world
state
and
previous
world
state

11. Undirected
model
for
chains
11
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
CompaObility
of
measurement
and
world
state
CompaObility
of
world
state
and
previous
world
state

12. Equivalence
of
chain
models
12
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Directed:
Undirected:
Equivalence:

13. Chain
model
for
sign
language
applicaOon
13
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
ObservaOons
are
normally
distributed
but
depend
on
sign
k
World
state
is
categorically
distributed,
parameters
depend
on
previous
world
state

14. Structure
• Chain
and
tree
models
• MAP
inference
in
chain
models
• MAP
inference
in
tree
models
• Maximum
marginals
in
chain
models
• Maximum
marginals
in
tree
models
• Models
with
loops
• ApplicaOons
14
14
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince

15. MAP
inference
in
chain
model
15
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
MAP
inference:
SubsOtuOng
in
:
Directed
model:

16. MAP
inference
in
chain
model
16
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Takes
the
general
form:
Unary
term:
Pairwise
term:

17. Dynamic
programming
17
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Maximizes
funcOons
of
the
form:
Set
up
as
cost
for
traversing
graph
–
each
path
from
le`
to
right
is
one
possible
configuraOon
of
world
states

18. Dynamic
programming
18
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Algorithm:
1. Work
through
graph
compuOng
minimum
possible
cost
to
reach
each
node
2. When
we
get
to
last
column,
find
minimum
3. Trace
back
to
see
how
we
got
there

19. Worked
example
19
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Unary
cost
Pairwise
costs:
• Zero
cost
to
stay
at
same
label
• Cost
of
2
to
change
label
by
1
• Infinite
cost
for
changing
by
more
than
one
(not
shown)

20. Worked
example
20
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Minimum
cost
to
reach
first
node
is
just
unary
cost

21. Worked
example
21
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Minimum
cost
is
minimum
of
two
possible
routes
to
get
here
Route
1:
2.0+0.0+1.1
=
3.1
Route
2:
0.8+2.0+1.1
=
3.9

22. Worked
example
22
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Minimum
cost
is
minimum
of
two
possible
routes
to
get
here
Route
1:
2.0+0.0+1.1
=
3.1
-­‐-­‐
this
is
the
minimum
–
note
this
down
Route
2:
0.8+2.0+1.1
=
3.9

23. Worked
example
23
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
General
rule:

24. Worked
example
24
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Work
through
the
graph,
compuOng
the
minimum
cost
to
reach
each
node

25. Worked
example
25
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Keep
going
unOl
we
reach
the
end
of
the
graph

26. Worked
example
26
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Find
the
minimum
possible
cost
to
reach
the
final
column

27. Worked
example
27
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Trace
back
the
route
that
we
arrived
here
by
–
this
is
the
minimum
configuraOon

28. Structure
• Chain
and
tree
models
• MAP
inference
in
chain
models
• MAP
inference
in
tree
models
• Maximum
marginals
in
chain
models
• Maximum
marginals
in
tree
models
• Models
with
loops
• ApplicaOons
28
28
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince

29. MAP
inference
for
trees
29
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince

30. MAP
inference
for
trees
30
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince

31. Worked
example
31
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince

32. Worked
example
32
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Variables
1-­‐4
proceed
as
for
the
chain
example.

33. Worked
example
33
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
At
variable
n=5
must
consider
all
pairs
of
paths
from
into
the
current
node.

34. Worked
example
34
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Variable
6
proceeds
as
normal.
Then
we
trace
back
through
the
variables,
splilng
at
the
juncOon.

35. Structure
• Chain
and
tree
models
• MAP
inference
in
chain
models
• MAP
inference
in
tree
models
• Maximum
marginals
in
chain
models
• Maximum
marginals
in
tree
models
• Models
with
loops
• ApplicaOons
35
35
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Extra
Extra
Jump
there

36. Marginal
posterior
inference
36
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
• Start
by
compuOng
the
marginal
distribuOon
over
the
Nth
variable
• Then
we`ll
consider
how
to
compute
the
other
marginal
distribuOons

37. CompuOng
one
marginal
distribuOon
37
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Compute
the
posterior
using
Bayes`
rule:
We
compute
this
expression
by
wriOng
the
joint
probability
:

38. CompuOng
one
marginal
distribuOon
38
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Problem:
CompuOng
all
NK
states
and
marginalizing
explicitly
is
intractable.
SoluOon:
Re-­‐order
terms
and
move
summaOons
to
the
right

39. CompuOng
one
marginal
distribuOon
39
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Define
funcOon
of
variable
w1
(two
rightmost
terms)
Then
compute
funcOon
of
variables
w2
in
terms
of
previous
funcOon
Leads
to
the
recursive
relaOon

40. CompuOng
one
marginal
distribuOon
40
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
We
work
our
way
through
the
sequence
using
this
recursion.
At
the
end
we
normalize
the
result
to
compute
the
posterior
Total
number
of
summaOons
is
(N-­‐1)K
as
opposed
to
KN
for
brute
force
approach.

41. Forward-­‐backward
algorithm
41
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
• We
could
compute
the
other
N-­‐1
marginal
posterior
distribuOons
using
a
similar
set
of
computaOons
• However,
this
is
inefficient,
as
much
of
the
computaOon
is
duplicated
• The
forward-­‐backward
algorithm
computes
all
of
the
marginal
posteriors
at
once
SoluOon:
Compute
all
first
term
using
a
recursion
Compute
all
second
terms
using
a
recursion
...
and
take
products

42. Forward
recursion
42
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Using
condiOonal
independence
relaOons
CondiOonal
probability
rule
This
is
the
same
recursion
as
before

43. Backward
recursion
43
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Using
condiOonal
independence
relaOons
CondiOonal
probability
rule
This
is
another
recursion
of
the
form

44. Forward
backward
algorithm
44
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Compute
the
marginal
posterior
distribuOon
as
product
of
two
terms
Forward
terms:
Backward
terms:

45. Belief
propagaOon
45
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
• Forward
backward
algorithm
is
a
special
case
of
a
more
general
technique
called
belief
propagaOon
• Intermediate
funcOons
in
forward
and
backward
recursions
are
considered
as
messages
conveying
beliefs
about
the
variables.
• We’ll
examine
the
Sum-­‐Product
algorithm.
• The
sum-­‐product
algorithm
operates
on
factor
graphs.

46. Sum
product
algorithm
46
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
• Forward
backward
algorithm
is
a
special
case
of
a
more
general
technique
called
belief
propagaOon
• Intermediate
funcOons
in
forward
and
backward
recursions
are
considered
as
messages
conveying
beliefs
about
the
variables.
• We’ll
examine
the
Sum-­‐Product
algorithm.
• The
sum-­‐product
algorithm
operates
on
factor
graphs.

47. Factor
graphs
47
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
• One
node
for
each
variable
• One
node
for
each
funcOon
relaOng
variables

48. Sum
product
algorithm
48
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Forward
pass
• Distribute
evidence
through
the
graph
Backward
pass
• Collates
the
evidence
Both
phases
involve
passing
messages
between
nodes:
• The
forward
phase
can
proceed
in
any
order
as
long
as
the
outgoing
messages
are
not
sent
unOl
all
incoming
ones
received
• Backward
phase
proceeds
in
reverse
order
to
forward

49. Sum
product
algorithm
49
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Three
kinds
of
message
• Messages
from
unobserved
variables
to
funcOons
• Messages
from
observed
variables
to
funcOons
• Messages
from
funcOons
to
variables

50. Sum
product
algorithm
50
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Message
type
1:
• Messages
from
unobserved
variables
z
to
funcOon
g
• Take
product
of
incoming
messages
• InterpretaOon:
combining
beliefs
Message
type
2:
• Messages
from
observed
variables
z
to
funcOon
g
• InterpretaOon:
conveys
certain
belief
that
observed
values
are
true

51. Sum
product
algorithm
51
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Message
type
3:
• Messages
from
a
funcOon
g
to
variable
z
• Takes
beliefs
from
all
incoming
variables
except
recipient
and
uses
funcOon
g
to
a
belief
about
recipient
CompuOng
marginal
distribuOons:
• A`er
forward
and
backward
passes,
we
compute
the
marginal
dists
as
the
product
of
all
incoming
messages

52. Sum
product:
forward
pass
52
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Message
from
x1
to
g1:
By
rule
2:

53. Sum
product:
forward
pass
53
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Message
from
g1
to
w1:
By
rule
3:

54. Sum
product:
forward
pass
54
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Message
from
w1
to
g1,2:
By
rule
1:
(product
of
all
incoming
messages)

55. Sum
product:
forward
pass
55
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Message
from
g1,2
from
w2:
By
rule
3:

56. Sum
product:
forward
pass
56
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Messages
from
x2
to
g2
and
g2
to
w2:

57. Sum
product:
forward
pass
57
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Message
from
w2
to
g2,3:
The
same
recursion
as
in
the
forward
backward
algorithm

58. Sum
product:
forward
pass
58
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Message
from
w2
to
g2,3:

59. Sum
product:
backward
pass
59
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Message
from
wN
to
gN,N-1:

60. Sum
product:
backward
pass
60
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Message
from
gN,N-1
to
wN-1:

61. Sum
product:
backward
pass
61
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Message
from
gn,n-1
to
wn-1:
The
same
recursion
as
in
the
forward
backward
algorithm

62. Sum
product:
collaOng
evidence
62
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
• Marginal
distribuOon
is
products
of
all
messages
at
node
• Proof:

63. Structure
• Chain
and
tree
models
• MAP
inference
in
chain
models
• MAP
inference
in
tree
models
• Maximum
marginals
in
chain
models
• Maximum
marginals
in
tree
models
• Models
with
loops
• ApplicaOons
63
63
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince

64. Marginal
posterior
inference
for
trees
64
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Apply
sum-­‐product
algorithm
to
the
tree-­‐structured
graph.

65. Structure
• Chain
and
tree
models
• MAP
inference
in
chain
models
• MAP
inference
in
tree
models
• Maximum
marginals
in
chain
models
• Maximum
marginals
in
tree
models
• Models
with
loops
• ApplicaOons
65
65
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince

66. Tree
structured
graphs
66
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
This
graph
contains
loops
But
the
associated
factor
graph
has
structure
of
a
tree
Can
sOll
use
Belief
PropagaOon

67. Learning
in
chains
and
trees
67
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Supervised
learning
(where
we
know
world
states
wn)
is
relaOvely
easy.
Unsupervised
learning
(where
we
do
not
know
world
states
wn)
is
more
challenging.
Use
the
EM
algorithm:
• E-­‐step
–
compute
posterior
marginals
over
states
• M-­‐step
–
update
model
parameters
For
the
chain
model
(hidden
Markov
model)
this
is
known
as
the
Baum-­‐Welch
algorithm.

68. Grid-­‐based
graphs
68
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
O`en
in
vision,
we
have
one
observaOon
associated
with
each
pixel
in
the
image
grid.

69. Why
not
dynamic
programming?
69
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
When
we
trace
back
from
the
final
node,
the
paths
are
not
guaranteed
to
converge.

70. Why
not
dynamic
programming?
70
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince

71. Why
not
dynamic
programming?
71
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
But:

72. Approaches
to
inference
for
grid-­‐based
models
72
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
1. Prune
the
graph.
Remove
edges
unOl
an
edge
remains

73. 73
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
2.
Combine
variables.
Merge
variables
to
form
compound
variable
with
more
states
unOl
what
remains
is
a
tree.
Not
pracOcal
for
large
grids
Approaches
to
inference
for
grid-­‐based
models

74. 74
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Approaches
to
inference
for
grid-­‐based
models
3.
Loopy
belief
propagaOon.
Just
apply
belief
propagaOon.
It
is
not
guaranteed
to
converge,
but
in
pracOce
it
works
well.
4.
Sampling
approaches
Draw
samples
from
the
posterior
(easier
for
directed
models)
5. Other
approaches
• Tree-­‐reweighted
message
passing
• Graph
cuts

75. Structure
• Chain
and
tree
models
• MAP
inference
in
chain
models
• MAP
inference
in
tree
models
• Maximum
marginals
in
chain
models
• Maximum
marginals
in
tree
models
• Models
with
loops
• ApplicaOons
75
75
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince

76. 76
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Gesture
Tracking

77. Stereo
vision
77
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
• Two
image
taken
from
slightly
different
posiOons
• Matching
point
in
image
2
is
on
same
scanline
as
image
1
• Horizontal
offset
is
called
disparity
• Disparity
is
inversely
related
to
depth
• Goal
–
infer
dispariOes
wm,n
at
pixel
m,n
from
images
x(1)
and
x(2)
Use
likelihood:

78. Stereo
vision
78
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince

79. Stereo
vision
79
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
1.
Independent
pixels

80. Stereo
vision
80
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
2.
Scanlines
as
chain
model
(hidden
Markov
model)

81. Stereo
vision
81
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
3.
Pixels
organized
as
tree
(from
Veksler
2005)

82. Pictorial
Structures
82
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince

83. SegmentaOon
83
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince

84. Part
1
Conclusion
84
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
• For
the
special
case
of
chains
and
trees
we
can
perform
MAP
inference
and
compute
marginal
posteriors
efficiently.
• Unfortunately,
many
vision
problems
are
defined
on
pixel
grids
–
this
requires
special
methods

85. Part
2:
Graphical
Models
for
Grids
85

86. 86
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
• Stereo
vision
Example
ApplicaOon

87. Part
2
Structure
• Denoising
problem
• Markov
random
fields
(MRFs)
• Max-­‐flow
/
min-­‐cut
• Binary
MRFs
-­‐
submodular
(exact
soluOon)
• MulO-­‐label
MRFs
–
submodular
(exact
soluOon)
• MulO-­‐label
MRFs
-­‐
non-­‐submodular
(approximate)
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
87

88. Models
for
grids
88
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
• Consider
models
with
one
unknown
world
state
at
each
pixel
in
the
image
–
takes
the
form
of
a
grid.
• Loops
in
the
graphical
model,
so
cannot
use
dynamic
programming
or
belief
propagaOon
• Define
probability
distribuOons
that
favor
certain
configuraOons
of
world
states
– Called
Markov
random
fields
– Inference
using
a
set
of
techniques
called
graph
cuts

89. 89
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Binary
Denoising
Before
A`er
Image
represented
as
binary
discrete
variables.
Some
proporOon
of
pixels
randomly
changed
polarity.

90. 90
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
MulO-­‐label
Denoising
Before
A`er
Image
represented
as
discrete
variables
represenOng
intensity.
Some
proporOon
of
pixels
randomly
changed
according
to
a
uniform
distribuOon.

91. 91
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Denoising
Goal
Observed
Data
Uncorrupted
Image

92. • Most
of
the
pixels
stay
the
same
• Observed
image
is
not
as
smooth
as
original
Now
consider
pdf
over
binary
images
that
encourages
smoothness
–
Markov
random
field
92
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Denoising
Goal
Observed
Data
Uncorrupted
Image

93. 93
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Markov
random
fields
This
is
just
the
typical
property
of
an
undirected
model.
We’ll
conOnue
the
discussion
in
terms
of
undirected
models

94. 94
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Markov
random
fields
Normalizing
constant
(parOOon
funcOon)
PotenOal
funcOon
Returns
posiOve
number
Subset
of
variables
(clique)

95. 95
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Markov
random
fields
Normalizing
constant
(parOOon
funcOon)
Cost
funcOon
Returns
any
number
Subset
of
variables
(clique)
RelaOonship

96. 96
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Smoothing
Example

97. 97
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Smoothing
Example
Smooth
soluOons
(e.g.
0000,1111)
have
high
probability
Z
was
computed
by
summing
the
16
un-­‐normalized
probabiliOes

98. 98
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Smoothing
Example
Samples
from
larger
grid
-­‐-­‐
mostly
smooth
Cannot
compute
parOOon
funcOon
Z
here
-­‐
intractable

99. 99
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Denoising
Goal
Observed
Data
Uncorrupted
Image

100. 100
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Denoising
overview
Bayes’
rule:
Likelihoods:
Prior:
Markov
random
field
(smoothness)
MAP
Inference:
Graph
cuts
Probability
of
flipping
polarity

101. 101
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Denoising
with
MRFs
Observed
image,
x
Original
image,
w
MRF
Prior
(pairwise
cliques)
Inference
:
Likelihoods

102. MAP
Inference
102
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Unary
terms
(compatability
of
data
with
label
y)
Pairwise
terms
(compatability
of
neighboring
labels)

103. 103
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Graph
Cuts
Overview
Unary
terms
(compatability
of
data
with
label
y)
Pairwise
terms
(compatability
of
neighboring
labels)
Graph
cuts
used
to
opOmise
this
cost
funcOon:
Three
main
cases:

104. 104
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Graph
Cuts
Overview
Unary
terms
(compatability
of
data
with
label
y)
Pairwise
terms
(compatability
of
neighboring
labels)
Graph
cuts
used
to
opOmise
this
cost
funcOon:
Approach:
Convert
minimizaOon
into
the
form
of
a
standard
CS
problem,
MAXIMUM
FLOW
or
MINIMUM
CUT
ON
A
GRAPH
Polynomial-­‐Ome
methods
for
solving
this
problem
are
known

105. 105
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Max-­‐Flow
Problem
Goal:
To
push
as
much
‘flow’
as
possible
through
the
directed
graph
from
the
source
to
the
sink.
Cannot
exceed
the
(non-­‐negaOve)
capaciOes
cij
associated
with
each
edge.

106. 106
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Saturated
Edges
When
we
are
pushing
the
maximum
amount
of
flow:
• There
must
be
at
least
one
saturated
edge
on
any
path
from
source
to
sink
(otherwise
we
could
push
more
flow)
• The
set
of
saturated
edges
hence
separate
the
source
and
sink

107. 107
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
AugmenOng
Paths
Two
numbers
represent:
current
flow
/
total
capacity

108. Choose
any
route
from
source
to
sink
with
spare
capacity,
and
push
as
much
flow
as
you
can.
One
edge
(here
6-­‐t)
will
saturate.
108
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
AugmenOng
Paths

109. 109
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
AugmenOng
Paths
Choose
another
route,
respecOng
remaining
capacity.
This
Ome
edge
6-­‐5
saturates.

110. 110
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
AugmenOng
Paths
A
third
route.
Edge
1-­‐4
saturates

111. 111
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
AugmenOng
Paths
A
fourth
route.
Edge
2-­‐5
saturates

112. 112
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
AugmenOng
Paths
A
fi`h
route.
Edge
2-­‐4
saturates

113. There
is
now
no
further
route
from
source
to
sink
–
there
is
a
saturated
edge
along
every
possible
route
(highlighted
arrows)
113
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
AugmenOng
Paths

114. 114
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
AugmenOng
Paths
The
saturated
edges
separate
the
source
from
the
sink
and
form
the
min-­‐cut
soluOon.
Nodes
either
connect
to
the
source
or
connect
to
the
sink.

115. Graph
Cuts:
Binary
MRF
Unary
terms
(compatability
of
data
with
label
w)
Pairwise
terms
(compatability
of
neighboring
labels)
Graph
cuts
used
to
opOmise
this
cost
funcOon:
First
work
with
binary
case
(i.e.
True
label
w
is
0
or
1)
Constrain
pairwise
costs
so
that
they
are
“zero-­‐diagonal”
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
115

116. Graph
ConstrucOon
• One
node
per
pixel
(here
a
3x3
image)
• Edge
from
source
to
every
pixel
node
• Edge
from
every
pixel
node
to
sink
• Reciprocal
edges
between
neighbours
Note
that
in
the
minimum
cut
EITHER
the
edge
connecOng
to
the
source
will
be
cut,
OR
the
edge
connecOng
to
the
sink,
but
NOT
BOTH
(unnecessary).
Which
determines
whether
we
give
that
pixel
label
1
or
label
0.
Now
a
1
to
1
mapping
between
possible
labelling
and
possible
minimum
cuts
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
116

117. Graph
ConstrucOon
Now
add
capaciOes
so
that
minimum
cut,
minimizes
our
cost
funcOon
Unary
costs
U(0),
U(1)
avached
to
links
to
source
and
sink.
• Either
one
or
the
other
is
paid.
Pairwise
costs
between
pixel
nodes
as
shown.
• Why?
Easiest
to
understand
with
some
worked
examples.
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
117

118. Example
1
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
118

119. Example
2
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
119

120. Example
3
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
120

121. Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
121

122. Graph
Cuts:
Binary
MRF
Unary
terms
(compatability
of
data
with
label
w)
Pairwise
terms
(compatability
of
neighboring
labels)
Graph
cuts
used
to
opOmise
this
cost
funcOon:
Summary
of
approach
• Associate
each
possible
soluOon
with
a
minimum
cut
on
a
graph
• Set
capaciOes
on
graph,
so
cost
of
cut
matches
the
cost
funcOon
• Use
augmenOng
paths
to
find
minimum
cut
• This
minimizes
the
cost
funcOon
and
finds
the
MAP
soluOon
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
122

123. General
Pairwise
costs
Modify
graph
to
• Add
P(0,0)
to
edge
s-­‐b
• Implies
that
soluOons
0,0
and
1,0
also
pay
this
cost
• Subtract
P(0,0)
from
edge
b-­‐a
• SoluOon
1,0
has
this
cost
removed
again
Similar
approach
for
P(1,1)
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
123

124. ReparameterizaOon
The
max-­‐flow
/
min-­‐cut
algorithms
require
that
all
of
the
capaciOes
are
non-­‐negaOve.
However,
because
we
have
a
subtracOon
on
edge
a-­‐b
we
cannot
guarantee
that
this
will
be
the
case,
even
if
all
the
original
unary
and
pairwise
costs
were
posiOve.
The
soluOon
to
this
problem
is
reparamaterizaOon:
find
new
graph
where
costs
(capaciOes)
are
different
but
choice
of
minimum
soluOon
is
the
same
(usually
just
by
adding
a
constant
to
each
soluOon)
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
124

125. ReparameterizaOon
1
The
minimum
cut
chooses
the
same
links
in
these
two
graphs
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince

126. ReparameterizaOon
2
The
minimum
cut
chooses
the
same
links
in
these
two
graphs
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
126

127. Submodularity
Adding
together
implies
Subtract
constant
β
Add
constant,
β
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
127

128. Submodularity
If
this
condiOon
is
obeyed,
it
is
said
that
the
problem
is
“submodular”
and
it
can
be
solved
in
polynomial
Ome.
If
it
is
not
obeyed
then
the
problem
is
NP
hard.
Usually
it
is
not
a
problem
as
we
tend
to
favour
smooth
soluOons.
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
128

129. Denoising
Results
Original
Pairwise
costs
increasing
Pairwise
costs
increasing
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
129

130. Plan
of
Talk
• Denoising
problem
• Markov
random
fields
(MRFs)
• Max-­‐flow
/
min-­‐cut
• Binary
MRFs
–
submodular
(exact
soluOon)
• MulO-­‐label
MRFs
–
submodular
(exact
soluOon)
• MulO-­‐label
MRFs
-­‐
non-­‐submodular
(approximate)
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
130

131. ConstrucOon
for
two
pixels
(a
and
b)
and
four
labels
(1,2,3,4)
There
are
5
nodes
for
each
pixel
and
4
edges
between
them
have
unary
costs
for
the
4
labels.
One
of
these
edges
must
be
cut
in
the
min-­‐cut
soluOon
and
the
choice
will
determine
which
label
we
assign.
MulOple
Labels
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
131

132. Constraint
Edges
The
edges
with
infinite
capacity
poinOng
upwards
are
called
constraint
edges.
They
prevent
soluOons
that
cut
the
chain
of
edges
associated
with
a
pixel
more
than
once
(and
hence
given
an
ambiguous
labelling)
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
132

133. MulOple
Labels
Inter-­‐pixel
edges
have
costs
defined
as:
Superfluous
terms
:
For
all
i,j
where
K
is
number
of
labels
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
133

134. Example
Cuts
Must
cut
links
from
before
cut
on
pixel
a
to
a`er
cut
on
pixel
b.
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
134

135. Pairwise
Costs
If
pixel
a
takes
label I and
pixel
b
takes
label
J
Must
cut
links
from
before
cut
on
pixel
a
to
a`er
cut
on
pixel
b.
Costs
were
carefully
chosen
so
that
sum
of
these
links
gives
appropriate
pairwise
term.
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
135

136. ReparameterizaOon
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
136

137. Submodularity
We
require
the
remaining
inter-­‐pixel
links
to
be
posiOve
so
that
or
By
mathemaOcal
inducOon
we
can
get
the
more
general
result
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
137

138. Submodularity
If
not
submodular,
then
the
problem
is
NP
hard.
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
138

139. Convex
vs.
non-­‐convex
costs
QuadraOc
• Convex
• Submodular
Truncated
QuadraOc
• Not
Convex
• Not
Submodular
Povs
Model
• Not
Convex
• Not
Submodular
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
139

140. What
is
wrong
with
convex
costs?
• Pay
lower
price
for
many
small
changes
than
one
large
one
• Result:
blurring
at
large
changes
in
intensity
Observed
noisy
image
Denoised
result
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
140

141. Plan
of
Talk
• Denoising
problem
• Markov
random
fields
(MRFs)
• Max-­‐flow
/
min-­‐cut
• Binary
MRFs
-­‐
submodular
(exact
soluOon)
• MulO-­‐label
MRFs
–
submodular
(exact
soluOon)
• MulO-­‐label
MRFs
-­‐
non-­‐submodular
(approximate)
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
141

142. Alpha
Expansion
Algorithm
• break
mulOlabel
problem
into
a
series
of
binary
problems
• at
each
iteraOon,
pick
label
α
and
expand
(retain
original
or
change
to
α)
IniOal
labelling
IteraOon
1
(orange)
IteraOon
3
(red)
IteraOon
2
(yellow)
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
142

143. Alpha
Expansion
Ideas
• For
every
iteraOon
– For
every
label
– Expand
label
using
opOmal
graph
cut
soluOon
Co-­‐ordinate
descent
in
label
space.
Each
step
opOmal,
but
overall
global
maximum
not
guaranteed
Proved
to
be
within
a
factor
of
2
of
global
opOmum.
Requires
that
pairwise
costs
form
a
metric:
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
143

144. Alpha
Expansion
ConstrucOon
Binary
graph
cut
–
either
cut
link
to
source
(assigned
to
α)
or
to
sink
(retain
current
label)
Unary
costs
avached
to
links
between
source,
sink
and
pixel
nodes
appropriately.
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
144

145. Alpha
Expansion
ConstrucOon
Graph
is
dynamic.
Structure
of
inter-­‐pixel
links
depends
on
α
and
the
choice
of
labels.
There
are
four
cases.
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
145

146. Alpha
Expansion
ConstrucOon
Case
1:
Adjacent
pixels
both
have
label
α
already.
Pairwise
cost
is
zero
–
no
need
for
extra
edges.
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
146

147. Alpha
Expansion
ConstrucOon
Case
2:
Adjacent
pixels
are
α,β.
Result
either
• α,α
(no
cost
and
no
new
edge).
• α,β
(P(α,β),
add
new
edge).
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
147

148. Alpha
Expansion
ConstrucOon
Case
3:
Adjacent
pixels
are
β,β.
Result
either
• β,β
(no
cost
and
no
new
edge).
• α,β
(P(α,β),
add
new
edge).
• β,α
(P(β,α),
add
new
edge).
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
148

149. Alpha
Expansion
ConstrucOon
Case
4:
Adjacent
pixels
are
β,γ.
Result
either
• β,γ
(P(β,γ),
add
new
edge).
• α,γ
(P(α,γ),
add
new
edge).
• β,α
(P(β,α),
add
new
edge).
• α,α
(no
cost
and
no
new
edge).
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
149

150. Example
Cut
1
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
150

151. Example
Cut
1
Important!
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
151

152. Example
Cut
2
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
152

153. Example
Cut
3
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
153

154. Denoising
Results
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
154

155. 155
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
CondiOonal
Random
Fields

156. 156
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Directed
model
for
grids
Cannot
use
graph
cuts
as
three-­‐wise
term.
Easy
to
draw
samples.

157. 157
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
• Background
subtracOon
ApplicaOons

158. 158
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
• Grab
cut
ApplicaOons

159. 159
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
• Shi`-­‐map
image
ediOng
ApplicaOons

160. 160
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince

161. 161
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
• Shi`-­‐map
image
ediOng
ApplicaOons

162. 162
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
• Super-­‐resoluOon
ApplicaOons

163. 163
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
• Texture
synthesis
ApplicaOons

164. 164
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
Image
QuilOng

165. 165
Computer
vision:
models,
learning
and
inference.
©2011
Simon
J.D.
Prince
• Synthesizing
faces
ApplicaOons

166. Further
resources
• hvp://www.computervisionmodels.com/
– Code
– Links
+
readings
(for
these
and
other
topics)
• Conference
papers
online:
BMVC,
CVPR,
ECCV,
ICCV,
etc.
• Jobs
mailing
lists:
Imageworld,
Visionlist
166