Course: Intro to Computer Science (Malmö Högskola): A overview of computability and complexity (for non-mathematicians). definition of algorithm, turing machines, lambda, calculus and concepts of complexity

1. Algorithms,
Computability
and
Complexity
Overview of mathematical concepts
Review of concepts already discussed

2. Algorithm
• You have a task
• You want to formulate an algorithm to
perform the task
• Three basic questions:
– What Is an algorithm?
– Will an algorithm be able to compute the task?
– How long will the algorithm take?
• how complex is the algorithm?

3. Three questions
about algorithms
• These are actually non trivial questions
– What Is an algorithm?
– Will an algorithm be able to compute the task?
– How long will the algorithm take?
• how complex is the algorithm?
• Important to mathematically formalize these
questions
– to create better algorithms
– To know where the limitations of algorithms lie

4. Questions
• What Is an algorithm?
– To make an algorithm, you have to know “exactly” what it
is.. Its components and how to put these components
together.
– To ask questions, you have to know what you are asking
questions about
• Will an algorithm be able to compute the task?
– Is the task I want impossible?
– Is it possible to formulate an impossible task
– What is the consequences for other algorithms
• How complex is the algorithm?
• Will it take “too much” time? (for cryptography, this is a good
thing)
• Can I make a faster algorithm for a given type of task?

5. DEFINITION
What Is an algorithm?
To make an algorithm, you have to know “exactly” what it is.. Its components
and how to put these components together.
To ask questions, you have to know what you are asking questions about

6. Algorithm Definition
• Part of the research is to formulate an exact
workable definition of an algorithm
• There are several/many definitions
– The do have common characteristics

7. Algorithm Definition
• A logical sequence of steps for solving a
problem, …
» From http://Dictionary.msn.com
• Dale and Lewis:
» a plan of solution for a problem
» Algorithm – An unambiguous (and precise) set of steps
for solving a problem (or sub-problem) in a finite amount
of time using a finite amount of data.

8. Algorithm Definition
• Shackelford, Russell L. in Introduction to
Computing and Algorithms –
– “An algorithm is a specification of a behavioral
process. It consists of a finite set of instructions
that govern behavior step-by-step.”

9. Definition
• An algorithm is a finite sequence of step by
step, discrete, unambiguous instructions for
solving a particular problem
– has input data, and is expected to produce output
data
– each instruction can be carried out in a finite
amount of time in a deterministic way

10. Keywords
• Notice the term finite.
– Algorithms should lead to an eventual solution.
– The algorithm should stop (halt).
• Sequence of steps
– Each step should do one logical action.

11. COMPUTABILITY
Will an algorithm be able to compute the task?
Is the task I want impossible?
Is it possible to formulate an impossible task
What is the consequences for other algorithms

12. Computability vs Complexity
• Computability refers to whether of not in
principle it is possible to evaluate f(n) by
following a set of instructions.
• We are not for the moment worried if this
computation requires 1,000,000 or more
consecutive steps.
• The latter refers to complexity which we will
return to later.

13. Ensheidungsproblem
Decision problem
Hilbert 1928
The Entscheidungsproblem asks for
an algorithm
that takes as input a statement of a first-order logic
and
answers "Yes" or "No"
according to whether the statement is universally valid
(valid in every structure satisfying the axioms).
An algorithm to ask whether something is true
Is it computable

14. Ensheidungsproblem
Decision problem
Hilbert 1928
• Before the question could be answered, the
notion of "algorithm" had to be formally
defined.
– This was done by Alonzo Church in 1936 with the
concept of "effective calculability" based on his λ
calculus
– and by Alan Turing in the same year with his
concept of Turing machines.
– It was recognized immediately by Turing that
these are equivalent models of computation.

15. 15
Alan Turing
(1912 – 1954)
Alonzo Church
(1903-1995)
Turing Machine Lambda calculus
Two mathematical ways to ask questions about
“computability”

16. Equivalent Computers
z z zz z z z
1
Start
HALT
), X, L
2:
look
for (
#, 1, -
), #, R
(, #, L
(, X, R
#, 0, -
Finite State Machine
...
Turing Machine

term = variable
| term term
| (term)
|  variable . term
y. M  v. (M [y v])
where v does not occur in M.
(x. M)N   M [ x N ]


Lambda Calculus

17. Alan Turing, 1936
• "On computable numbers, with an application
to the Entscheidungsproblem [decision
problem]”
• addressed a previously unsolved
mathematical problem posed by the German
mathematician David Hilbert in 1928:

18. Turing, 1936
Is there, in principal,
any definite mechanical method or process
by which
all mathematical questions
could be decided?
The Decision Problem

19. Turing, 1936
• To answer the question (in the negative),
• Turing proposed a simple abstract computing
machine,
• modeled after a mathematician with a pencil,
an eraser, and a stack of pieces of paper
• He asserted that any function is a computable
function if it is computable by one of these
abstract machines.

20. Turing, 1936
• He then investigated whether there were
some mathematical problems which could not
be solved by any such abstract machine.
• Such abstract computing machines are now
called "Turing machines".
• One particular Turing machine, called a
"Universal Turing Machine", served as the
model for designing actual programmable
computers.

21. Why a Turing Machine
Because Turing machines are very simple
compared with computers in practical use,
conceptually easier
to prove impossibility results
for Turing machines.
These impossibility results
apply as well to all known computers

22. Turing, 1936
• Startling result of Turing's 1936 paper
• assertion that there are well-defined
problems that cannot be solved by any
computational procedure.

23. What is Computable?
Computation is usually modelled as a mapping from inputs to outputs,
carried out by a formal “machine,” or program, which processes its
input in a sequence of steps.
An “effectively computable” function is one that can be computed in
a finite amount of time using finite resources.
…………
…………
…
input
yes
no
output
Effectively
computable
function

24. Unsolvable
• problems are formulated as functions, we call
unsolvable functions are
• noncomputable;
• Problems formulated as predicates,
• they are called
• undecidable.
• Using Turing's concept of the abstract machine,
we would say that a function is noncomputable if
there exists no Turing machine that could
compute it.

25. 25
............
Tape
Read-Write head
Control Unit

26. -Dept. of CSIE, CCU, Taiwan- 26
............
Read-Write head
No boundaries -- infinite length
The head moves Left or Right
The
tape
OR

27. -Dept. of CSIE, CCU, Taiwan- 27
............
Read-Write head
The head at each time step:
1. Reads a symbol
2. Writes a symbol
3. Moves Left or Right

28. -Dept. of CSIE, CCU, Taiwan- 28
............
Example:
Time 0
............
Time 1
a a cb
a b k c
1. Reads a
2. Writes k
3. Moves Left
a

29. -Dept. of CSIE, CCU, Taiwan- 29
............
Time 1
a b k c
............
Time 2
a k cf
1. Reads b
2. Writes f
3. Moves Right
b

30. Add 1 to a sequence of ones
If 1, then move right If 1, then move left
Loop
If 0, replace with 1
Start: 0 0 0 1 1 1 1 0 0 0 4 ones in a row
End: 0 0 0 1 1 1 1 1 0 0 5 ones in a row

31. So what?
You can add a 1 to a sequence of ones
From a set of a few (a manageable number) of primitive operations
If a number is represented by a sequence of ones
then you have the x+1 operator, i.e addition
This is a/(the most) basic mathematical operator
Powerful enough to represent complex mathematics
You can derive other (all) mathematic operations and objects

32. Powerful “language”
From a set of a few (a manageable number) of
primitive operations
Easier enough
To formulate
Proofs about algorithms
Powerful enough
To
represent complex
mathematics
All the ingredients are there Only have to deal with a few basic operations
Not worried about complexity…. Worried about, for example, “can” we compute?

33. -Dept. of CSIE, CCU, Taiwan- 33
The Input String
............    
Blank symbol
head
a b ca
Head starts at the leftmost position
of the input string
Input string

34. -Dept. of CSIE, CCU, Taiwan- 34
States & Transitions
1q 2qLba ,
Read Write Move Left
1q 2qRba ,
Move Right

35. -Dept. of CSIE, CCU, Taiwan- 35
Turing machine for the language }{ nn
ba
0q 1q 2q3q
Rxa ,
Raa ,
Ryy ,
Lyb ,
Laa ,
Lyy ,
Rxx ,
Ryy ,
Ryy ,
4q
L,

36. -Dept. of CSIE, CCU, Taiwan- 36
0q 1q 2q3q
Rxa ,
Raa ,
Ryy ,
Lyb ,
Laa ,
Lyy ,
Rxx ,
Ryy ,
Ryy ,
4q
L,
 ba
0q
a bTime 0 

37. -Dept. of CSIE, CCU, Taiwan- 37
0q 1q 2q3q
Rxa ,
Raa ,
Ryy ,
Lyb ,
Laa ,
Lyy ,
Rxx ,
Ryy ,
Ryy ,
4q
L,
 bx
1q
a b Time 1

38. -Dept. of CSIE, CCU, Taiwan- 38
0q 1q 2q3q
Rxa ,
Raa ,
Ryy ,
Lyb ,
Laa ,
Lyy ,
Rxx ,
Ryy ,
Ryy ,
4q
L,
 x
2q
a b Time 2 b

39. -Dept. of CSIE, CCU, Taiwan- 39
0q 1q 2q3q
Rxa ,
Raa ,
Ryy ,
Lyb ,
Laa ,
Lyy ,
Rxx ,
Ryy ,
Ryy ,
4q
L,
 yx
2q
a b Time 3

40. -Dept. of CSIE, CCU, Taiwan- 40
0q 1q 2q3q
Rxa ,
Raa ,
Ryy ,
Lyb ,
Laa ,
Lyy ,
Rxx ,
Ryy ,
Ryy ,
4q
L,
 yx
2q
a b Time 4

41. -Dept. of CSIE, CCU, Taiwan- 41
0q 1q 2q3q
Rxa ,
Raa ,
Ryy ,
Lyb ,
Laa ,
Lyy ,
Rxx ,
Ryy ,
Ryy ,
4q
L,
 yx
0q
a b Time 5

42. -Dept. of CSIE, CCU, Taiwan- 42
0q 1q 2q3q
Rxa ,
Raa ,
Ryy ,
Lyb ,
Laa ,
Lyy ,
Rxx ,
Ryy ,
Ryy ,
4q
L,
 yx
1q
x b Time 6

43. -Dept. of CSIE, CCU, Taiwan- 43
0q 1q 2q3q
Rxa ,
Raa ,
Ryy ,
Lyb ,
Laa ,
Lyy ,
Rxx ,
Ryy ,
Ryy ,
4q
L,
 yx
1q
x b Time 7

44. -Dept. of CSIE, CCU, Taiwan- 44
0q 1q 2q3q
Rxa ,
Raa ,
Ryy ,
Lyb ,
Laa ,
Lyy ,
Rxx ,
Ryy ,
Ryy ,
4q
L,
 yx x y
2q
Time 8

45. -Dept. of CSIE, CCU, Taiwan- 45
0q 1q 2q3q
Rxa ,
Raa ,
Ryy ,
Lyb ,
Laa ,
Lyy ,
Rxx ,
Ryy ,
Ryy ,
4q
L,
 yx x y
2q
Time 9

46. -Dept. of CSIE, CCU, Taiwan- 46
0q 1q 2q3q
Rxa ,
Raa ,
Ryy ,
Lyb ,
Laa ,
Lyy ,
Rxx ,
Ryy ,
Ryy ,
4q
L,
 yx
0q
x y Time 10

47. -Dept. of CSIE, CCU, Taiwan- 47
0q 1q 2q3q
Rxa ,
Raa ,
Ryy ,
Lyb ,
Laa ,
Lyy ,
Rxx ,
Ryy ,
Ryy ,
4q
L,
 yx
3q
x y Time 11

48. -Dept. of CSIE, CCU, Taiwan- 48
0q 1q 2q3q
Rxa ,
Raa ,
Ryy ,
Lyb ,
Laa ,
Lyy ,
Rxx ,
Ryy ,
Ryy ,
4q
L,
 yx
3q
x y Time 12

49. -Dept. of CSIE, CCU, Taiwan- 49
0q 1q 2q3q
Rxa ,
Raa ,
Ryy ,
Lyb ,
Laa ,
Lyy ,
Rxx ,
Ryy ,
Ryy ,
4q
L,
 yx
4q
x y 
Halt & Accept
Time 13

50. What is Computable?
Computation is usually modelled as a mapping from inputs to outputs,
carried out by a formal “machine,” or program, which processes its
input in a sequence of steps.
An “effectively computable” function is one that can be computed in
a finite amount of time using finite resources.
…………
…………
…
input
yes
no
output
Effectively
computable
function

51. Uncomputable
Using Turing's concept of the abstract machine,
we would say that a function
is
noncomputable
if there exists
no Turing machine that could compute it.

52. Computability
Function f is computable
if some program P computes it:
For any input x,
the computation P(x)
halts
with output f(x)
Finite number of steps

53. Algorithm for 𝚷
• To calculate the area of a circle,
• we need the value of 𝚷
• Can we find an algorithm to compute 𝚷?

54. 𝚷
• 3.14159265358979323846264338327950288419716939937510
• 58209749445923078164062862089986280348253421170679
• 82148086513282306647093844609550582231725359408128
• 48111745028410270193852110555964462294895493038196
• 44288109756659334461284756482337867831652712019091
• 45648566923460348610454326648213393607260249141273
• 72458700660631558817488152092096282925409171536436
• 78925903600113305305488204665213841469519415116094
• 33057270365759591953092186117381932611793105118548
• 07446237996274956735188575272489122793818301194912
• 98336733624406566430860213949463952247371907021798
• 60943702770539217176293176752384674818467669405132
• 00056812714526356082778577134275778960917363717872
• 14684409012249534301465495853710507922796892589235
• 42019956112129021960864034418159813629774771309960
• 51870721134999999837297804995105973173281609631859
• 50244594553469083026425223082533446850352619311881
• 71010003137838752886587533208381420617177669147303
• 59825349042875546873115956286388235378759375195778
• 18577805321712268066130019278766111959092164201989…….
The first 1000 digits of 𝚷
Goes on as long as you want…
Π 10,000 digits
Π 100,000 digits

55. Algorithm for 𝚷
An algorithm (in general)
is
a finite sequence
of step by step, discrete, unambiguous
instructions for solving a particular problem
Problem
𝚷 is a infinite sequence of numbers…
cannot calculate with a finite number of steps
But there are algorithms????

56. Calculating 𝚷
• We can’t
– Calculate the EXACT value of 𝚷
• However, we can
– Calculate 𝚷 to whatever accuracy we desire
– Finite number of digits, finite number of steps

57. Probabilistic Algorithms
Calculate pi with a dart board
Area of square
d2
Area of Circle:
P
d
2
æ
è
ç
ö
ø
÷
2
prob =
circle
square
=
p
d
2
æ
è
ç
ö
ø
÷
2
d2
=
p
4
Probability dart will be in circle
d
number darts in circle
divided by
number of darts in total
times
Is π
Monte Carlo Method
Always Gives an answer
But not necessarily Correct
The probability of correctness goes up with time

58. Calculate
All do not stop (halt)…..
They can be an algorithm if only an approximation is needed

59. Halting function
• Decide whether program halts on input
– Given program P and input x to P,
Halt (P,x) =
Fact: There is no algorithm for Halt
yes if P(x) halts
no otherwise
Clarifications
Assume program P requires one string input x
Write P(x) for output of P when run in input x
Program P is string input to Halt

60. Unsolvability of the halting problem
• Suppose P solves variant of halting problem
– On input Q, assume
P(Q) =
• Build program D
– D(Q) =
• Does this make sense? What can D(D) do?
– If D(D) halts, then D(D) runs forever.
– If D(D) runs forever, then D(D) halts.
– CONTRADICTION: program P must not exist.
yes if Q(Q) halts
no otherwise
run forever if Q(Q) halts
halt if Q(Q) runs forever

61. Lambda Calculus
Alonzo Church
(1903-1995)
Another formulation
of
computability

62. Equivalent Computers
z z zz z z z
1
Start
HALT
), X, L
2:
look
for (
#, 1, -
), #, R
(, #, L
(, X, R
#, 0, -
Finite State Machine
...
Turing Machine

term = variable
| term term
| (term)
|  variable . term
y. M  v. (M [y v])
where v does not occur in M.
(x. M)N   M [ x N ]


Lambda Calculus

63. What is Calculus?
• In High School:
d/dx xn = nxn-1 [Power Rule]
d/dx (f + g) = d/dx f + d/dx g [Sum Rule]
Calculus is a branch of mathematics that
deals with limits and the differentiation and
integration of functions of one or more
variables...

64. Real Definition
• A calculus is just a bunch of rules for
manipulating symbols.
• People can give meaning to those symbols,
but that’s not part of the calculus.
• Differential calculus is a bunch of rules for
manipulating symbols. There is an
interpretation of those symbols corresponds
with physics, slopes, etc.

65. -calculus
Alonzo Church, 1940
term = variable
| term term
| (term)
|  variable . term

66. -calculus
 variable . term
F(variable) = term

67. Why?
• Once we have precise and formal rules for
manipulating symbols, we can use it to reason
with.
• Since we can interpret the symbols as
representing computations, we can use it to
reason about programs.

68. Universal Computer
• Lambda Calculus can simulate a Turing Machine
– Everything a Turing Machine can compute, Lambda
Calculus can compute also
• Turing Machine can simulate Lambda Calculus
– Everything Lambda Calculus can compute, a Turing
Machine can compute also
• Church-Turing Thesis: this is true for any other
mechanical computer also

69. Normal Steps
• Turing machine:
– Read one square on tape, follow one FSM
transition rule, write one square on tape, move
tape head one square
• Lambda calculus:
– One beta reduction
• Your PC:
– Execute one instruction (?)
• What one instruction does varies

70. -Reduction
(the source of all computation)
(x. M)N
This is the function definition
M is an expression with the variable x
N is used at the value of x
N substitutes x in the expression M

71. (x. x)a = a
-Reduction
(the source of all computation)
Identity operation
Function gives the argument as output
(x. xx)a = aa
(y.(x. xy)a)b =(y.ay)b=ab
(y.(x. xy) z.zy)b =(y.(z.zy)y)b
= (y. yy)b
=bb

72. -calculus
Useful and expression enough that a programming
language was modeled after it
LISP
was developed from -calculus
not the other way round.)
This was the “first” general language of
Artificial intelligence
Also
The “first” functional programming language

73. Complexity
Algorithm:
An algorithm is a finite sequence of step by step, discrete,
unambiguous instructions for solving a particular problem
Computable:
Does an algorithm exist??
Complexity:
The algorithm exists
How “complex” is it?
How many “descrete steps” does it have?
how much does each step cost?
How does complexity depend on size of input?

74. Vector/Array of size n
Efficient for the nth element
A single arithmetic calculation
Complexity does not increase as the vector gets bigger
O(1)
76 8 9 10 11 120 1 2 3 4 5
Pos(v[0]) Pos(v[0]) + 5
Find ith element of vector
This is exactly an example for what a vector is designed for

75. Cost/Complexity
Compute: Pos(v[0]) + n
Find the nth element
Cost:
C1: Cost of determining the address of the first element Pos(v[0])
Does not depend on the size of the array of numbers
C2: Cost of adding n to this address… cost of addition operation
Does not depend on the size of the array of numbers
Total cost: C1 + C2

76. Cost/Complexity
Compute: Pos(v[0]) + nFind the nth element
Cost: Does not depend on the size of the array of numbers
Total cost = T = C1 + C2
N: size of array
T:TotalCost
Cost(n) = T a constant amount
In the O notation
O(1)
The constant factor is not so important
It is only important that there in no
dependence on the size of the array

77. Vector/Array of size n
Insert before ith element in vector
76 8 9 10 11 120 1 2 3 4 5
1. Allocate vector of size n+1
2. Copy element 0 to i-1 to places 0 to i-1 and
copy elements i to n-1 to places i+1 to n
76 8 9 10 11 120 1 2 3 4 5
1 operation
n operations
3. Set in element
76 8 9 10 11 120 1 2 3 4 5 13
1 operation
Vectors are not designed to be used with insertion operations
As the vector gets larger, the insertion takes more time/operations
O(n)

78. Cost of insertion
• C1: Allocate array size n+1
• C2: Copy I elements from old to new array
• C2=(i)*c (c: the cost to copy an element)
• C3: Copy the element to insert
• C3= c (c: the cost to copy an element)
• C4: Copy n-I elements from old to new array
• C4=(n-i)*c
Insert a new element at position I in array of length n
Total Cost = T = C1 + C2 + C3 + C4
= C1 + i*c + c + (n-i)*c
= C1 + (n+1)*c
Total Cost = T = C1 + (n+1)*c

79. Cost of insertion
Insert a new element at position I in array of length n
Total Cost = T(n) = C1 + (n+1)*c
Cost: Depends on the size of the array of numbers
N: size of array
T:TotalCost
Cost(n) = C1 + (n+1)*c
In the O notation
O(n)
The constant factor is not so important
It is only important that there in no
dependence on the size of the array

80. Complexity
O(1) Time/operation complexity does not increase
with the size of the problem
O(n) Time/operation complexity does increases linearly
with the size of the problem
Find ith element of vector
Insert before ith element in vector
N: size of array
T:TotalCost
N: size of array
T:TotalCost

81. Complexity of Bubble Sort

82. Bubble Sort: Idea
• Idea: bubble in water.
– Bubble in water moves upward. Why?
• How?
– When a bubble moves upward, the water
from above will move downward to fill in
the space left by the bubble.

83. Bubble Sort Example
9, 6, 2, 12, 11, 9, 3, 7
6, 9, 2, 12, 11, 9, 3, 7
6, 2, 9, 12, 11, 9, 3, 7
6, 2, 9, 12, 11, 9, 3, 7
6, 2, 9, 11, 12, 9, 3, 7
6, 2, 9, 11, 9, 12, 3, 7
6, 2, 9, 11, 9, 3, 12, 7
6, 2, 9, 11, 9, 3, 7, 12The 12 is greater than the 7 so they are exchanged.
The 12 is greater than the 3 so they are exchanged.
The twelve is greater than the 9 so they are exchanged
The 12 is larger than the 11 so they are exchanged.
In the third comparison, the 9 is not larger than the 12 so no
exchange is made. We move on to compare the next pair without
any change to the list.
Now the next pair of numbers are compared. Again the 9 is the
larger and so this pair is also exchanged.
Bubblesort compares the numbers in pairs from left to right
exchanging when necessary. Here the first number is compared
to the second and as it is larger they are exchanged.
The end of the list has been reached so this is the end of the first pass. The
twelve at the end of the list must be largest number in the list and so is now in
the correct position. We now start a new pass from left to right.

84. Bubble Sort Example
6, 2, 9, 11, 9, 3, 7, 122, 6, 9, 11, 9, 3, 7, 122, 6, 9, 9, 11, 3, 7, 122, 6, 9, 9, 3, 11, 7, 122, 6, 9, 9, 3, 7, 11, 12
6, 2, 9, 11, 9, 3, 7, 12
Notice that this time we do not have to compare the last two
numbers as we know the 12 is in position. This pass therefore only
requires 6 comparisons.
First Pass
Second Pass

85. Bubble Sort Example
2, 6, 9, 9, 3, 7, 11, 122, 6, 9, 3, 9, 7, 11, 122, 6, 9, 3, 7, 9, 11, 12
6, 2, 9, 11, 9, 3, 7, 12
2, 6, 9, 9, 3, 7, 11, 12
Second Pass
First Pass
Third Pass
This time the 11 and 12 are in position. This pass therefore only
requires 5 comparisons.

86. Bubble Sort Example
2, 6, 9, 3, 7, 9, 11, 122, 6, 3, 9, 7, 9, 11, 122, 6, 3, 7, 9, 9, 11, 12
6, 2, 9, 11, 9, 3, 7, 12
2, 6, 9, 9, 3, 7, 11, 12
Second Pass
First Pass
Third Pass
Each pass requires fewer comparisons. This time only 4 are needed.
2, 6, 9, 3, 7, 9, 11, 12Fourth Pass

87. Bubble Sort Example
2, 6, 3, 7, 9, 9, 11, 122, 3, 6, 7, 9, 9, 11, 12
6, 2, 9, 11, 9, 3, 7, 12
2, 6, 9, 9, 3, 7, 11, 12
Second Pass
First Pass
Third Pass
The list is now sorted but the algorithm does not know this until it
completes a pass with no exchanges.
2, 6, 9, 3, 7, 9, 11, 12Fourth Pass
2, 6, 3, 7, 9, 9, 11, 12Fifth Pass

88. Bubble Sort Example
2, 3, 6, 7, 9, 9, 11, 12
6, 2, 9, 11, 9, 3, 7, 12
2, 6, 9, 9, 3, 7, 11, 12
Second Pass
First Pass
Third Pass
2, 6, 9, 3, 7, 9, 11, 12Fourth Pass
2, 6, 3, 7, 9, 9, 11, 12Fifth Pass
Sixth Pass
2, 3, 6, 7, 9, 9, 11, 12
This pass no exchanges are made so the algorithm knows the list is
sorted. It can therefore save time by not doing the final pass. With
other lists this check could save much more work.

89. Bubble Sort: Example
• Notice that at least one element will be
in the correct position each iteration.
40 2 1 43 3 65 0 -1 58 3 42 4
652 1 40 3 43 0 -1 58 3 42 4
65581 2 3 40 0 -1 43 3 42 4
1 2 3 400 65-1 43 583 42 4
1
2
3
4

90. 1 0 -1 32 653 43 5842404
Bubble Sort: Example
0 -1 1 2 653 43 58424043
-1 0 1 2 653 43 58424043
6
7
8
1 2 0 3-1 3 40 6543 584245

91. Bubble Sort Complexity
1 2 3 4 5 n-1 comparisons Total Cost= (n-1) * c
Even the lowest cost is a function of n
Cost = (n-1)*c = -c + n*c
O(n)
Remember: not so interested in the constant factors
N: size of array
T:TotalCost

92. Bubble Sort Complexity
5 4 3 2 1 Worse case: exactly opposite
4 5 3 2 1 Compare (c ) and swap (s)
4 3 5 2 1 Compare (c ) and swap (s)
4 3 2 5 1 Compare (c ) and swap (s)
4 3 2 1 5 Compare (c ) and swap (s)
3 2 1 4 5
3 4 2 1 5 Compare (c ) and swap (s)
3 2 4 1 5 Compare (c ) and swap (s)
Compare (c ) and swap (s)
1 2 3 4 5 Compare (c ) and swap (s)
Move 5 to place
4* (c+s)
Move 5 to place
3* (c+s)
Move 5 to place
1* (c+s)
2 3 1 4 5 Compare (c ) and swap (s)
2 1 3 4 5 Compare (c ) and swap (s)
Move 5 to place
2* (c+s)

93. Bubble Sort Complexity
Total Cost:
5*(c+s) + 4*(c+s) + 3*(c+s) + 2*(c+s) 1*(c+s)
(5+4+3+2+1)*(c+s)
15*(c+s)
3*5*(c+s)
In general the sum of numbers from 1 to n is:
If n odd: n*(n+1)/2
If n even n*n/2

94. Bubble Sort Complexity
Size of array to sort
Cost:
In general the sum of numbers
from 1 to n is:
If n odd: n*(n+1)/2= n2/2 + n/2
If n even n*n/2 = n2/2
Approx: n2/2
For a large n….
n2 is much much larger than n

95. Complexities
Size of object for algorithm
Cost:
O(n)
O(1)
O(n2)

96. Tractability
• Some problems are intractable:
as they grow large, we are unable to solve them
in reasonable time
• What constitutes reasonable time?
– Standard working definition: polynomial time
– On an input of size n the worst-case running time is
O(nk) for some constant k
– O(n2), O(n3), O(1), O(n lg n), O(2n), O(nn), O(n!)
– Polynomial time: O(n2), O(n3), O(1), O(n lg n)
– Not in polynomial time: O(2n), O(nn), O(n!)

97. Polynomial-Time Algorithms
• Are some problems solvable in polynomial time?
– Of course: many algorithms we’ve studied provide
polynomial-time solutions to some problems
• Are all problems solvable in polynomial time?
– No: Turing’s “Halting Problem” is not solvable by any
computer, no matter how much time is given
• Most problems that do not yield polynomial-time
algorithms are either optimization or decision
problems.

98. Optimization/Decision Problems
• An optimization problem tries to find an optimal solution
• A decision problem tries to answer a yes/no question
• Many problems will have decision and optimization versions
– Eg: Traveling salesman problem
• optimization: find hamiltonian cycle of minimum weight
• decision: is there a hamiltonian cycle of weight  k
• Some problems are decidable, but intractable:
as they grow large, we are unable to solve them in reasonable
time
– Is there a polynomial-time algorithm that solves the problem?

99. The Class P
P: the class of decision problems that have polynomial-time
deterministic algorithms.
– That is, they are solvable in O(p(n)), where p(n) is a polynomial on n
– A deterministic algorithm is (essentially) one that always computes the
correct answer
Why polynomial?
– if not, very inefficient
– nice closure properties
• the sum and composition of two polynomials are always polynomials too

100. Sample Problems in P
• Fractional Knapsack
• MST
• Sorting
• Others?

101. The class NP
NP: the class of decision problems that are solvable in polynomial
time on a nondeterministic machine (or with a nondeterministic
algorithm)
• (A determinstic computer is what we know)
• A nondeterministic computer is one that can “guess” the right
answer or solution
– Think of a nondeterministic computer as a parallel machine that can
freely spawn an infinite number of processes
• Thus NP can also be thought of as the class of problems
– whose solutions can be verified in polynomial time
• Note that NP stands for “Nondeterministic Polynomial-time”

102. Review: P And NP Summary
• P = set of problems that can be solved in polynomial
time
– Examples: Fractional Knapsack, …
• NP = set of problems for which a solution can be
verified in polynomial time
– Examples: Fractional Knapsack,…, Hamiltonian Cycle,
CNF SAT, 3-CNF SAT
• Clearly P  NP
• Open question: Does P = NP?
– Most suspect not
– An August 2010 claim of proof that P ≠ NP, by Vinay
Deolalikar, researcher at HP Labs, Palo Alto, has flaws

103. NP-complete problems
• A decision problem D is NP-complete iff
1. D  NP
2. every problem in NP is polynomial-time reducible
to D
• Cook’s theorem (1971): CNF-sat is NP-
complete

104. NP-Complete
“NP-Complete” comes from:
– Nondeterministic Polynomial
– Complete - “Solve one, Solve them all”
There are more NP-Complete problems than
provably intractable problems.