Presentation by Olaf Nimz at the Zurich R Meetup, January 8, 2018

1. BASLE BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA
HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH
Anomaly Detection
database integrated
Dr. Olaf Nimz

2. Our company.
anomaly detection2 08/01/2018
Trivadis is a market leader in IT consulting, system integration, solution engineering
and the provision of IT services focusing on and
technologies
in Switzerland, Germany, Austria and Denmark. We offer our services in the following
strategic business fields:
Trivadis Services takes over the interacting operation of your IT systems.
O P E R A T I O N

3. Scoring Engine for R
3
EXEC sp_execute_external_script
@language =N'R',
-- SQL Part (sends to @script)
@input_data_1 =N 'SELECT 1 as Installed',
-- R Part (gets @input_data_1)
@script=N'OutputDataSet<-InputDataSet'
WITH RESULT SETS
(([Installed] int not null));
GO
Microsoft R ServerLaunchpad
(BxlServer and
SQL Satellite,
Rserver.dll)
08/01/2018 anomaly detection

4. Agenda
anomaly detection4 08/01/2018
1. Prioritise data quality effort
Cleansing DWH
IoT Streams
Online Learning
2. Unsupervised Measures
Mahalanobis distance
Clustering
Local Outlier Factor
Isolation Forest
Variational AutoEncoders

5. Novelty, Noise, Outlier, Anomaly, Fraud, Instability
anomaly detection5 08/01/2018
1. Special Observations
in Relation to Baseline
Contextual
2. Suspicious Observations
Novelty
Outlier / Anomaly
Data quality issue
Instable Process
Random Noise

6. Local - Global
anomaly detection6 08/01/2018

7. Local - Global
anomaly detection7 08/01/2018
High dimensional: Distance is meaningless

8. 
Approach for Detection
anomaly detection8 08/01/2018
1. Statistical Distribution
Entropy
2. Deviation from Normal
Sequence of Events
Conditional (temporal, spatial context)
Collective like DoS-Attack
3. Distance to neighborhood
4. Local Density
5. High-dimensional adaptations
Subspace projection
Angle based
 



9. Univariate Extreme Values
anomaly detection9 08/01/2018
IQR = Inter quartile range
~ 95%
Median = 50% Percentile
50%
> 2 stdev (~ 2%)
Grubb’s test per point
Scaling by z-Scores
(robust using
Median absolute deviation)

10. Multivariate data
anomaly detection10 08/01/2018
Robust
Mahalanobis
Distance
chisq.plot()
dimensions
2

11. No outlier ?
anomaly detection11 08/01/2018
+

12. SCADA of Wind Turbine
anomaly detection16 08/01/2018

13. Power Curve – Deviation from Prediction
anomaly detection17 08/01/2018
lm( power ~ wind_speed
+ I(wind_speed^2)
+ I(wind_speed^3) , data)
R2
adj.= 95%
Sample

14. Boxplot – univariate
anomaly detection18 08/01/2018
Mean
Median
Power
WindSpeed
Temperature

15. Wind distribution
anomaly detection19 08/01/2018

16. Mahalanobis Distance
anomaly detection20 08/01/2018
Multivariate:
multi dimensional
Scale:
How many stdev
away from center ?
Mahalanobis Distance
candidates

17. Outlier in Orginal Space
anomaly detection21 08/01/2018

18. Eigenvector Space
anomaly detection22 08/01/2018

19. Overview of Outliers
anomaly detection23 08/01/2018
Leland Wilkinson's probabilistic HDoutlier model
=> for mixture of numeric and categorical variables
1D
2D
3D
4D

20. HDBSCAN
anomaly detection24 08/01/2018
Only Borderline & Core cases
Scaled by z-Scores

21. Local Outlier Factor
anomaly detection25 08/01/2018
Reachability distance: It can be "reached" from its neighbors.
LOF: relative Reachability compared to its neighbours
K-Nearest Neighbors
Reachability distance

22. Hierarchical Clustering
anomaly detection26 08/01/2018
MDS coloured by cuttree

23. Isolation Forest - Emsemble
anomaly detection27 08/01/2018

24. Challenges
anomaly detection28 08/01/2018
1. Manual threshold – automatic is expensive
2. Mixed data type (numeric & categorical)
3. High dimensional spaces
4. High Cardinality (Granularity)
5. Multi-Modal: Global vs Local Scope
6. Online
http://projects.rajivshah.com/shiny/outlier/