Approximation models (or surrogate models) provide an efficient substitute to expen- sive physical simulations and an efficient solution to the lack of physical models of system behavior. However, it is challenging to quantify the accuracy and reliability of such ap- proximation models in a region of interest or the overall domain without additional system evaluations. Standard error measures, such as the mean squared error, the cross-validation error, and the Akaikes information criterion, provide limited (often inadequate) informa- tion regarding the accuracy of the final surrogate. This paper introduces a novel and model independent concept to quantify the level of errors in the function value estimated by the final surrogate in any given region of the design domain. This method is called the Re- gional Error Estimation of Surrogate (REES). Assuming the full set of available sample points to be fixed, intermediate surrogates are iteratively constructed over a sample set comprising all samples outside the region of interest and heuristic subsets of samples inside the region of interest (i.e., intermediate training points). The intermediate surrogate is tested over the remaining sample points inside the region of interest (i.e., intermediate test points). The fraction of sample points inside region of interest, which are used as interme- diate training points, is fixed at each iteration, with the total number of iterations being pre-specified. The estimated median and maximum relative errors within the region of in- terest for the heuristic subsets at each iteration are used to fit a distribution of the median and maximum error, respectively. The estimated statistical mode of the median and the maximum error, and the absolute maximum error are then represented as functions of the density of intermediate training points, using regression models. The regression models are then used to predict the expected median and maximum regional errors when all the sample points are used as training points. Standard test functions and a wind farm power generation problem are used to illustrate the effectiveness and the utility of such a regional error quantification method.

1. Quantifying Regional Error in Surrogates by Modeling its
Relationship with Sample Density
Ali Mehmani, Souma Chowdhury , Jie Zhang, Weiyang Tong,
and Achille Messac
Syracuse University, Department of Mechanical and Aerospace Engineering
54th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and
Materials Conference
April 8-11, 2013, Boston, Massachusetts

2. Surrogate model
• Surrogate models are commonly used for providing a tractable and
inexpensive approximation of the actual system behavior in many
routine engineering analysis and design activities:
2

3. 3
Broad Research Question
Lower Fidelity How Low?
Structural Blade Design
ANSYS, Inc.
Surrogate Model
Expensive Inexpensive

4. 4
Broad Research Question
How to quantify the level of the surrogate accuracy ?
 further improvement of the surrogate,
 domain exploration,
 assessing the reliability of the optimal design,
 quantifying the uncertainty associated with the surrogate,
 construction of a weighted surrogate model, and
 …

5. Research Objective
 Develop a reliable method to quantify the surrogate error,
5
 This method should have the following characteristics:
 model independent
 no additional system evaluations
 local/global error measurement
 quantify the error of the actual surrogate

6. Regional Error Estimation of Surrogate
6

7. Regional Error Estimation of Surrogate
7
(REES)

8. Presentation Outline
• Review of surrogate model error measurement methods
8
• Relation of surrogate accuracy with sample density
• Regional Error Estimation of Surrogate
• Numerical examples: benchmark and an engineering
design problems

9. Presentation Outline
• Review of surrogate model error measurement methods
9
• Relation of surrogate accuracy with sample density
• Regional Error Estimation of Surrogate
• Numerical examples: benchmark and an engineering
design problems

10. Surrogate Model Error Measurement Methods
 Error quantification methods can be classified,
 based on their computational expense, into methods that require
additional data, and methods that use existing data.
 based on the region of interest, into:
- Global error measure
(e.g., split sample, cross-validation, Akaike’s information criterion, and
bootstrapping).
- Local or point-wise error measure
(e.g., the mean squared errors for Kriging and the linear reference model
(LRM).)
10

11. Surrogate Model Error Measurement Methods
 Error metrics,
11
• The mean squared error (MSE) (or root mean square error (RMSE) )
• The maximum absolute error (MAE)
• The relative absolute error (RAE)
actual values on ith test point
predicted values on ith test point

12. Surrogate Model Error Measurement Methods
 Error metrics,
• The prediction sum of square (PRESS) is based on the leave-one-out
12
cross-validation error
• The root mean square of PRESS (PRESSRMS) based on the k-fold cross-validation.
• The relative absolute error of cross-validation (RAECV) based on leave-one-
out approach.

13. Presentation Outline
• Review of surrogate model error measurement methods
13
• Relation of surrogate accuracy with sample density
• Regional Error Estimation of Surrogate
• Numerical examples: benchmark and an engineering
design problems

14. Methodology: Concept
14
Model accuracy ∝ Available resources
In general, this concept can be applied for different methodologies
- Surrogate modeling,
- Finite Element Analysis, and
- ...

15. Methodology: Concept
15
 Finite Element Analysis (numerical methods)
The finer mesh, the stresses are more precise due to the larger
number of elements
coarse mesh
(4 solid brick element)
medium mesh
(32 solid brick element)
fine mesh
(256 solid brick element)
Estimate total shear force and flexural moment at vertical
sections using Finite Element Analysis.

16. Methodology: Concept
 Surrogate (mathematical model)
Surrogate accuracy generally improves with increasing training points.
3 training points
9 training points
7 training points
The location of additional points has
strong impact on surrogate accuracy.
This impact is highly problem and model
dependent.
16

17. Presentation Outline
• Review of surrogate model error measurement methods
17
• Relation of surrogate accuracy with sample density
• Regional Error Estimation of Surrogate
• Numerical examples: benchmark and an engineering
design problems

18. Methodology: REES
 The REES method formulates the variation of error as a
18
function of training points using intermediate surrogates.
 This formulation is used to predict the level of error in a
final surrogate.

19. Methodology: REES
Step 1 : Generation of sample data
The entire set of sample points is represented by 푿 .
Step 2 : Identification of sample points inside/outside region of interest
{푿풊풏}
{푿풐풖풕}
푿 = 푿풊풏 + 푿풐풖풕
푿풊풏 : Inside-region data set
푿풐풖풕 : Outside-region data set
user-defined region of interest
19

20. Methodology: REES
Step 3 : Estimation of the variation of the error with sample density
Outside-Training region Point
point
Inside-region point
user-defined region
of interest
First Iteration :
Test Point
Second iteration :
Third iteration :
Final Surrogate :
Training Point

21. Methodology: REES
Step 3 : Estimation of the variation of the error with sample density
 A position of sample points which are selected as training
points, at each iteration, is critical to the surrogate accuracy.
 The proposed error measure should be minimally sensitive to
the location of the test points at each iteration.
21
 Intermediate surrogates are
iteratively constructed (at each
iteration) over a sample set
comprising all samples outside the
region of interest and heuristic
subsets of samples inside the region
of interest.

22. Methodology: REES
Step 3 : Estimation of the variation of the error with sample density
 The intermediate subset for each
 The number of iterations (푁푖푡) is defined
combination at specific iteration is defined
by
{휷풌} ⊂ 푿풊풏
#{휷풌} = 풏풕, 풏풕−ퟏ < 풏풕
풌 = 1,2, … , 퐾푡
- dimension of a problem,
- number of inside sample points, and
- preference of the user
 The intermediate training points and test
points for each combination at each
iteration is defined by
 The number of sample combinations
(푲풕 ) is defined,
푿푻푹 = 푿풐풖풕 + 휷풌
푿푻푬 = 푿 − 푿푻푹
 The intermediate surrogates
푓푘 , 풌 = ퟏ, ퟐ, . . , 푲풕
are constructed for all combinations using the
intermediate training points ( 푿푻푹 ), and are
tested over the intermediate test points ( 푿푻푬 ).

23. Methodology: REES
Step 3 : Estimation of the variation of the error with sample density
 The median and the maximum errors are
estimated for each combination
풎풕: the number of test points in tth iteration
풆: the RAE value estimated on intermediate test points
23

24. Methodology: REES
Step 3 : Estimation of the variation of the error with sample density
 The median and the maximum errors are
estimated for each combination
The median is a useful measures of central
tendency which is less vulnerable to outliers.
24
Median error
Overall Fidelity Information
Maximum error
Minimum Fidelity Information

25. Methodology: REES
Step 3 : Estimation of the variation of the error with sample density
 Probabilistic models are developed using
The mode of distribution is selected to
represent the errors at each iteration.
a lognormal distribution to represent
median and maximum errors estimated
over all 푲풕 combinations at each
iterations.
Mode of median error distribution
Mode of maximum error distribution
 These values are used to relate the
variation of the surrogate error with
number of training points (sample
density).

26. The relation of the error with sample density
 12-D Test Problem (Dixon & Price, n=12)
Number of sample points # 푿 = ퟓퟓퟎ, Number of inside sample points # 푿풊풏 = # 푿
Number of training points at each iteration,풏풕 = 5푡 + 50, 푡 = 1,2, … , 70
Number of sample combination, 푲풕 = 500
Number of Training Points
MOmax
# 푿푻푹 = ퟓퟓ
# 푿푻푬 = ퟒퟒퟓ
Number of Training Points
Estimated mode of median errors Estimated mode of maximum errors
MOmed
First iteration
Last iteration
# 푿푻푹 = ퟒퟎퟎ
# 푿푻푬 = ퟏퟎퟎ

27. The relation of the error with sample density
 12-D Test Problem (Dixon & Price, n=12)
Number of Training Points
Meanmean
Number of Training Points
Estimated mode of median errors Estimated mean of mean errors
MOmed
REES Method Normalized k-fold CV

28. Methodology: REES
Step 4 : Prediction of regional error in the final surrogate
 The final surrogate model is constructed using the full set of training data.
 Regression models are applied to relate
- the statistical mode of the median error distribution(푴풐풎풆풅)
- the statistical mode of the maximum error distributions(푴풐풎풂풙), and
- the absolute maximum error (푨푩푺풎풂풙)
at each iteration to the size of the inside-region training points (nt),
 These regression models are called the variation of error with sample density
(VESD).
The regression models are used to predict the level of the
error in the final surrogate within the region of interest.
28

29. Methodology: REES
Modeling the Variation of Regional Error with Training Point Density
 In this study, three types of the regression functions are used to represent
the variation of regional error with respect to the inside-region training points
Exponential regression model
Multiplicative regression model
Linear regression model
 The choice of these functions assume a smooth monotonic decrease of the
regional error with the training point density within that region.
 The root mean squared error metric is used to select the best-fit regression
model
29

30. Presentation Outline
• Review of surrogate model error measurement methods
30
• Relation of surrogate accuracy with sample density
• Regional Error Estimation of Surrogate
• Numerical examples: benchmark and an engineering
design problems

31. Numerical Examples
 The effectiveness of the REES method is explored for applications with
- Kriging,
- Radial Basis Functions (RBF),
- Extended Radial Basis Functions (E-RBF), and
- Quadratic Response Surface (QRS).
 To evaluate practical and numerical efficiencies of the REES method,
three benchmark problems and an engineering design problem are tested.
 The error evaluated using REES, and the relative absolute error given by
leave-one-out cross-validation (푹푨푬풄풗) are compared with the actual
error evaluated using relative absolute error on additional test
points (푹푨푬풂풄풕풖풂풍).
31

32. Median of RAEs
Numerical Examples
Results and Discussion
VESD regression models within the region of interest of surrogate models
constructed for the Branin-Hoo Function to predict,
Distribution of
median errors
Mode of the median error
distribution,
Predicted mode of median error
in the final surrogate,
VESDmed
Number of Inside-region Training Points
32

33. Numerical Examples
Results and Discussion
VESD regression models
within the region of interest of
surrogate models constructed
for the Branin-Hoo Function
to predict,
Type and coefficients of
VESDmed
Kriging RBF
E-RBF QRS

34. Maximum of RAEs
Numerical Examples
Results and Discussion
VESD regression models within the region of interest of surrogate models constructed for the
Branin-Hoo Function to predict the mode of maximum ( ) and the absolute
maximum ( ) error.
Distribution of
maximum errors
Mode of the maximum
error distribution,
Absolute maximum error
Predicted absolute
maximum error in
the final surrogate
Predicted mode of
maximum error in
the final surrogate
34 Number of Inside-region Training Points

35. Numerical Examples
Results and Discussion
VESD regression models within the region of interest of surrogate models constructed for the
Branin-Hoo Function to predict the mode of maximum ( ) and the absolute
maximum ( ) error.
Type and coefficients of VESDmax
Type and coefficients of VESDABS
Kriging RBF
E-RBF QRS

36. Numerical Examples
Wind Farm Power Generation
Surrogates are developed using Kriging, RBF, E-RBF, and QRS to
represent the power generation of an array-like wind farm.
36

37. Numerical Examples
Results and Discussion
VESD regression models in different surrogates for the wind farm power
generation problem
It. 1 It. 2 It. 3 It. 4 Predicted Error
37

38. Numerical Examples
38
Results and Discussion
The closer to one, the better the corresponding error measure.
predicted mode of median errors
median of RAEs evaluated on test
points
median of relative absolute
errors of cross-validation

39. Concluding Remarks
 We developed a new method to quantify surrogate error based on the
hypothesis that:
“The accuracy of the approximation model is related to the amount
of available resources”
 This relationship can be reliably quantified when the error measures is
less sensitive to sample locations or a type of application.
 The REES method addresses this issue.
 The preliminary results on benchmark and wind farm power generation
problems indicate that in majority of cases the REES method is more
accurate than other measures.
39
It is not possible using any existing methods

40. Future Works
 The scope for improvement the method
 The implementation of the proposed error measurement in
surrogate developments.
40

41. Acknowledgement
41
 I would like to acknowledge my research adviser
Prof. Achille Messac, and my co-adviser Prof.
Souma Chowdhury for their immense help and
support in this research.
 Support from the NSF Awards is also acknowledged.

42. 42
Thank you
Questions
and
Comments

43. Median of RAEs
Numerical Examples
Results and Discussion
VESD regression models within the region of interest of surrogate models
constructed for the Branin-Hoo Function to predict,
Distribution of
median errors
k-fold CV
Mode of the median error
distribution,
Predicted mode of median error
in the final surrogate,
VESDmed
Number of Inside-region Training Points
43
Mean mean
Number of Inside-region Training Points