The document discusses various optimization methods for solving different types of optimization problems. It begins by defining a general optimization problem and then describes several specific problem types including linear programming (LP), integer programming (IP), mixed-integer linear programming (MILP), nonlinear programming (NLP), and mixed-integer nonlinear programming (MINLP). It provides examples and discusses solution methods like the simplex algorithm, branch and bound, and decomposition approaches.
2. General optimization problem
x = vector of variables
f = objective function
g = constraints (either equalities or inequalities)
X = set of bounds/integrality constraint
𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝑜𝑟 𝑚𝑎𝑥𝑖𝑚𝑖𝑧𝑒 𝑓(𝑥)
𝑠. 𝑡. 𝑔 𝑥 ≤ 0
x ∈ 𝑋
5. Linear Programming (LP)
𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝐶 𝑇 𝑋
𝑠. 𝑡. 𝐴𝑋 − 𝐵 ≤ 0
𝑥𝑖 ≥ 0
Examples:
- Flux balance analysis (FBA) of
metabolic networks
- Flow through networks
Methods:
- SIMPLEX
Solvers:
- CPLEX (freely available from IBM for
academia in R, MATLAB, Python, GAMS)
6. Flux Balance Analysis
A + B 2C v1
C + B D v2
Biochemical reactions
r1 r2 rn
A -1 0 … 0
B -1 -1 … 0
C 2 -1 … 0
D 0 1 … -1
……………..
M 0 0 … 1
*
v1
v2
…
…
…
vn
0
0
…
…
…
0
=
Mathematical representation
Z= c1 v1 + c2 v2 + c3 v3 … s.t. additional constrains (e.g., available
metabolites in the media)
Cellular objective
e.g., pyruvate production
e.g., biomass production
e.g., estradiol production
Genome-wide
metabolism
Steady-state assumption
7. SIMPLEX ALGORITHM
See this link for a simple step procedure of Dantzig’s algorithm
http://www.ams.jhu.edu/~castello/625.414/Handouts/SimplexProcedure.pdf
9. Integer Programming (IP)
𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝐶 𝑇 𝑌
𝑠. 𝑡. 𝐴𝑌 − 𝐵 ≤ 0
𝑦𝑖 𝑎𝑟𝑒 𝑖𝑛𝑡𝑒𝑔𝑒𝑟𝑠
Examples:
- Capital budgeting (e.g., identification of the
best investments sets subject to their
availability, b)
- Scheduling (e.g., class, faculty and students to
minimize the no-first choice)
Methods:
- Integer cuts (not very much used for IP by
itself)
- Branch and bound (BB)
Solvers:
- CPLEX (freely available from IBM for academia
in R, MATLAB, Python, GAMS)
10. Branch and Bound (BB)
𝑚𝑎𝑥𝑖𝑚𝑖𝑧𝑒 5𝑥1 + 8𝑥2
𝑠. 𝑡. 𝑥1 +𝑥2 ≤ 6
5𝑥1 + 9𝑥2 ≤ 45
𝑥1, 𝑥2 𝑎𝑟𝑒 𝑖𝑛𝑡𝑒𝑔𝑒𝑟𝑠
1. Relaxation of IP = LP, upper limit IP
2. Branch over one of the variables and
solve the relax LP
3. Determine whether to branch or bound
From http://web.mit.edu/15.053/www/AMP-Chapter-09.pdf
11. Branch and Bound (BB)
𝑚𝑎𝑥𝑖𝑚𝑖𝑧𝑒 5𝑥1 + 8𝑥2
𝑠. 𝑡. 𝑥1 +𝑥2 ≤ 6
5𝑥1 + 9𝑥2 ≤ 45
𝑥1, 𝑥2 𝑎𝑟𝑒 𝑖𝑛𝑡𝑒𝑔𝑒𝑟𝑠
1. Relaxation of IP = LP, upper limit IP
2. Branch over one of the variables and
solve the relax LP
3. Determine whether to branch or
bound
- Bound if infeasible or less than
previous optimization results
From http://web.mit.edu/15.053/www/AMP-Chapter-09.pdf
12. Branch and Bound (BB)
𝑚𝑎𝑥𝑖𝑚𝑖𝑧𝑒 5𝑥1 + 8𝑥2
𝑠. 𝑡. 𝑥1 +𝑥2 ≤ 6
5𝑥1 + 9𝑥2 ≤ 45
𝑥1, 𝑥2 𝑎𝑟𝑒 𝑖𝑛𝑡𝑒𝑔𝑒𝑟𝑠
1. Relaxation of IP = LP, upper limit IP
2. Branch over one of the variables and
solve the relax LP
3. Determine whether to branch or
bound
- Bound if infeasible or less than
previous optimization results
From http://web.mit.edu/15.053/www/AMP-Chapter-09.pdf
13. Branch and Bound (BB)
𝑚𝑎𝑥𝑖𝑚𝑖𝑧𝑒 5𝑥1 + 8𝑥2
𝑠. 𝑡. 𝑥1 +𝑥2 ≤ 6
5𝑥1 + 9𝑥2 ≤ 45
𝑥1, 𝑥2 𝑎𝑟𝑒 𝑖𝑛𝑡𝑒𝑔𝑒𝑟𝑠
1. Relaxation of IP = LP,
upper limit IP
2. Branch over one of the
variables and solve the
relax LP
3. Determine whether to
branch or bound
From http://web.mit.edu/15.053/www/AMP-Chapter-09.pdf
22. Constrained NLP
- Karus-Kuhn-Tucker (KKT) Conditions
- KKT points x* can be a
minimum, maximum or
saddle point.
- KKT are used when
the NLP problem has
constraints
25. Convex functions and sets
Function f(x) is a convex function if for every pair of points in its domain X, the line
segment joining these two points is above the graph of the function, i.e., if for any
x1, x2 ∈ X and any λ ∈ 0,1 we have
𝑓 (𝜆𝑥1 1 − 𝜆 𝑥2 ≤ 𝜆𝑓 𝑥1 + 1 − 𝜆 𝑓(𝑥2)
Convex Non convex
26. Convex sets
Set C is a convex set if the line segment between any two points in C lies in C, i.e.,
if for any x1, x2 ∈ C and any 𝜃 ∈ 0,1 we have
𝜃𝑥1 + (1 − 𝜃)𝑥2 ∈ 𝐶
.
x1 .
x2
.
x1
.
x2
Convex Non convex
28. Branch and Bound (BB)
▪ Ravindran and Gupta (1985)
▪ Similar to the MILP branch-and-bound –
tree enumeration
▪ Suitable for problems with reasonable
number of integer variables and the NLP
relaxation is tight
Decomposition
• Outer-Approximation (OA) Duran &
Grossmann (1986)
• Generalized Benders Decomposition (GBD),
Geoffrion (1972)
• Extended Cutting Plane (ECP), Westerlund
and Pettersson (1995)
Solution algorithms for Convex MINLP
29. Branch and Bound for Convex
MINLP
• Similar to the branch-and-bound for solving MILP
• Difference is that in each node we solve a relaxed NLP problem (relaxing the
integer variables in MINLP); nonlinear program (NLP) relaxation – Lower bound
• Less stringent than other methods
• Successive solution of NLP subproblems
• Advantage: Tight formulation may require one NLP
• Disadvantage: Potentially many NLP subproblems
• Convergence global optimum: Unique solution of NLP (sufficient condition)
𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝑓 𝑥, 𝑦
𝑠. 𝑡. 𝑔 𝑥, 𝑦 ≤ 0
ℎ 𝑥, 𝑦 = 0
𝑥, 𝑦 ≤ 0
Relax NLP
y are continuous variables
31. Integer cut
The choice of all 0-1 combinations for 𝑦𝑗, 𝑗 ∈ 𝐽 is feasible, except the ones
for which 𝑦𝑗 = 0, 𝑗 ∈ 𝑁, 𝑦𝑗 = 1, 𝑗 ∈ 𝐵, where N and B are specified
partitions of J
32. NLP Upper Limit
NLP-U: nonlinear program at fixed yk Upper bound
Solution of NLP-U is a valid upper bound of MINLP solution
33. Master MILP problem
Develop a equivalent linear representation of the MINLP and apply
relaxation
Assume functions are convex and differentiable
34. Outer approximation
𝑓 𝑥 ≥ 𝑓 𝑥 𝑘
+ 𝛻𝑓 𝑥 𝑘 𝑇
(x − 𝑥 𝑘
)
𝑔 𝑥 ≥ 𝑔 𝑥 𝑘
+ 𝛻𝑔 𝑥 𝑘 𝑇
(x − 𝑥 𝑘
)
As f(x) and g(x) are convex, their corresponding outer approximations
for a feasible solution 𝑥 𝑘 ∈ 𝑋 can be modeled by
f(x)
Underestimate objective
function z
g(x)
Overestimated of
the feasible region
35. Outer approximation
Based on the solution of the NLP-U (xk, yk), we add the outer approximation cuts for
f(x) and g(x)
37. Generalized Benders
decomposition
Consider the Outer-Approximation (xk, yk)
Obtain a linear combination of (1) using the Karush-Kuhn-Tucker multipler μk
Similarly, cuts for infeasible subproblems can be derived
39. Extending cutting plane
1. Guess (x0, y0)
2. Find xk, yk from the Master MILP from the Outer
Approximation
3. If 𝑔 𝑥 𝑘 + 𝐵𝑦 𝑘 ≤ 𝜀 => STOP
4. Otherwise k=k+1, linearize function at xk, yk and go
to step 2
No NLP
41. Convex vs Non-convex MINLP
Convex MINLP
▪ Branch and Bound
▪ Outer Approximation
▪ Generalized Benders Decomposition
▪ Extended Cutting Plane
▪ Major (local) MINLP solvers
Nonconvex MINLP - Global Optimization
▪ Convexification and Reformulation - nonconvex problems are much
more difficult to solve than convex ones
▪ Convex Envelopes
▪ Spatial Branch and Bound
▪ (Global) optimization solvers
42. Mixed Integer Non-Linear
Programming (MINLP)
𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝑓(𝑥, 𝑦)
𝑠. 𝑡. 𝑔(𝑥, 𝑦) ≤ 0
Examples:
- Modularity maximization
- Minimization of plant cost (fixed and variable) – chemical plant design
Methods:
- Convexification and Reformulation
- Convex Envelopes
- Spatial Branch and Bound
Solvers:
𝑥𝑖 ≥ 0
𝑦𝑗 𝑎𝑟𝑒 𝑖𝑛𝑡𝑒𝑔𝑒𝑟𝑠
- α BB, BARON (Branch and Reduce), OA for nonconvex MINLP, Branch
and Contract
43. Non-convexities
Effects of nonconvexities
▪ NLP supbroblems may have local
optima
▪ MILP master may cut-off global
optimum
Handling of nonconvexitieis
▪ Rigorous approach (global optimization)
- Replace nonconvex terms
- Using underestimators/convex envelopes
- Solve convex MINLP within spatial branch &
bound
Objective
Multiple minima
0 1
y
x
Global optimum
Cut off!
44. Convexification and Reformulation
Some nonconvex optimization problems can be reformulated as convex problem
(and solve with the convex optimization algorithms) – Convexification
A common case: Posynominals (ci are non-negative)
𝑓 𝑥 =
𝑖
𝑐𝑖
𝑗
𝑥𝑗
𝛼 𝑖𝑗
𝑥𝑗=𝑒 𝑢 𝑗
Replace the posynomials by an exponential function
45. Convex Envelopes
Most deterministic global optimization algorithms are based on spatial branch and
bound method
Branch-and-Bound for
MILP
Spatial Branch-and-Bound
for Nonconvex MINLP
Relaxation
Replacing binary/integer
variables as continuous
variables with bounds,
solve LPs
Replacing nonconvex terms
with their convex envelopes,
solve convex (MI)NLP
Branching
Branch binary/integer
variables only
Branch on both integer
variables and continuous
variables (due to the nonconvex
terms)
Complexity
NP-hard, but the worst case
can be bounded by 2n
NP-hard, much more difficult
48. McCormick envelopes for bilinear
terms
Convex envelopes of bilinear terms
McCormick under and over estimators
Over-estimators
Under-estimators
49. Other envelopes
If g(x) is twice derivable then the convex relation for g(x):
𝑐𝑜𝑛 𝑔 𝑥 = 𝑔 𝑥 +
𝑖
(𝑥𝑖
𝐿
− 𝑥𝑖)(𝑥𝑖 − 𝑥𝑖
𝑈
)
50. Step 1: Generate subproblems by branching on continuous
variables (subregions)
Step 2: Compute lower bound from convex MINLP (relaxation)
Step 3: Compute upper bound from local optimal solution to
nonconvex MINLP
Step 4: Continue until tolerance of bounds within tolerance
Spatial Branch and Bound Method
x
y Convex Relaxation
x
y
x
y
x
y
50
54. Modularity maximization - MIQP
Objective function
Constraints
Each node, n, belongs to only one module
Each link, l, should belong to the same
module, m, that the two nodes its belong,
n, e
Number of links in each module m
Total degree in each module m
There are more constraints in the paper…
57. Resources
Take the class ChE345: Process Optimization
GAMS, software for optimization
IBM academic initiative to obtain CPLEX
Some online resources:
1. Fenqi Yu webpage
https://optimization.mccormick.northwestern.edu/index.php/Main_Page
2. MIT optimization
http://web.mit.edu/15.053/www/AMP-Chapter-02.pdf
3. Argonne MINLP optimization
https://wiki.mcs.anl.gov/leyffer/images/b/b2/MINLP-Survey.pdf
