SlideShare uma empresa Scribd logo
1 de 29
© ETH Zürich | ICT-Networks/NSG christian.hallqvist@id.ethz.ch 27.08.2015
Simple Anomaly Detection
via Netflows
2
The Big Question:
What Is Under The Radar?
First Goal (1)
To create an automated method of detecting
unusual connections and/or anomalies with
netflows, thereby finding compromised hosts
3
Top Down Method Works Well With
IDS
4
N=100
P2P BitTorrent transfer „Deviation“
(total traffic)
Netflow Anomaly Detection
Top Down Strategy
Analyzing the total traffic down to individual hosts
by detecting behavioral deviations.
Been there and done that with netflows (2008).
The problem:
Even though a malicious traffic event is usually
an anomaly, an anomaly is not always a malicious
event
5
With an automated method, finding the
correlating netflows of incidents regardless of
source of information (IDS, Switch, AV, User,
Admin, Netflows)
New Goal (2)
6
Netflow Anomaly Detection
Bottom up Strategy
Viewing what an individual host is doing
compared to the general population
7
Common Problems Regardless of
Source of Information
• Vague Indications
• Detected Anomalies
• Recurring Compromise
• False Negatives
• False Positives
8
google
facebook
youtube
InternetETH
Usual Connections
9
google
facebook
youtube
???????
InternetETH
Unusual Connections
???????Causing IDS anomalies
10
google
facebook
youtube
???????
InternetETH
Correlations/Validation
s
???????Causing IDS anomalies
???????
??????? 11
Pros and Cons With This Method
Pro: It is automatic and indeed sometimes
successfull
Con: It may take a long time to run
Again:
All anomalies are not malicious.
The potential problem is when the individual host is
generating harmless but very diverse unusal traffic.
Both pros and cons:
It is possible to automatically sort the connections
based on how usual/unusal they are
12
google
facebook
youtube
?
InternetETH
Unusal Connections
?
Causing IDS
anomalies ?
?
?
?
?
?
?
?
13
A Real Example (Switch)
[SWITCH-CERT #22814x ]
Most likely compromised system
[129.132.208.10x]
[Botnet]
Based on received information about a
‘malicious IRC command master at
183.203.15.205
2013-08-14 17:40:08.070 14
./analyzerdynamic2.sh 129.132.208.10x 20130814 1800
Enter Comment. End it with ^D
Subject: [SWITCH-CERT #228144] Most likely compromised system [129.132.208.10x ...]
[Botnet]
Done. Comment stored in -rw-r--r-- 1 hall nsg 93 Aug 22 18:05 comment
DEBUG: starthour:201308141700 endhour:201308141800 startday:201308131800
port:-1
First DYNAMIC STAGE *************************************
nfdump -M /nfsen -R nfcapd.201308141700:nfcapd.201308141800 '( host
129.132.208.10x )' | grep ^2013 | awk '{ print $7 }' | sed s/:/ /g | awk '{ print $1 }' | grep -v
"129.132.208.10x:" | sort -u > ./analyzer_129.132.208.10x.outp Debug dnumber:41 Second
DYNAMIC STAGE *************************************
107.21.234.205
112 107.21.234.205 (40)
108.160.162.53
1938 108.160.162.53 (39)
108.160.163.46
229 108.160.163.46 (38)
12.130.131.80
120 12.130.131.80 (37)
... 15
…
120 121 12.130.131.80 10 5.0000000 .8264462
112 113 107.21.234.205 11 4.5454545 .8849557
95 96 77.67.22.188 2 .7142857 1.0416666
45 46 62.146.92.202 12 1.090909 2.17391304
34 35 137.205.124.72 3 .2497918 2.8571428
29 30 54.224.40.139 10 4.8309178 3.3333333
27 28 134.60.1.5 13 .0336482 3.5714285
23 24 88.156.222.90 4 1.8691588 4.1666666
17 18 178.63.20.18 5 2.6041666 5.5555555
7 8 193.36.36.16 91 5.6627255 12.5000000
3 3 183.203.8.238 1 .0959692 33.3333333
2 3 183.203.15.205 34 3.333333 33.333333316
List Result
Numberofips
Numberofips+
investigateddest
ip
Investigated
destip
Numberof
flows
Flowsin%
(1/Numberof
ips+
investigated
destip)*100
Blacklist Check Routine
77.67.22.188
62.146.92.202
137.205.124.72
54.224.40.139
134.60.1.5
88.156.222.90
178.63.20.18
193.36.36.16
183.203.8.238
183.203.15.205
botcc:
183.203.15.205
17
183.203.8.238
InternetETH
Further
Correlations/Validation
s
Can Now be Done
183.203.15.205
18
129.132.208.10x
The Concept in a Nutshell
1. Find all connections around the investigated
IP over a 60 minute period
2. Take those connections and rate how usual
(or unusal) these are in the general
population over a 24hr period
19
Find All Connections Around the Investigated
IP Over a 60 Minute Period
20
Time point
of interest
for the
investigated
ip
Internet
Investigated
IP at ETH
network
Take Those Connections and Rate How Usual
(or Unusal) These are in the General
Population Over a 24hr Period
InternetETH network
Another Real Example (IDS)
IDS Event with destination google:
EVENT:
ET TROJAN Zeus Bot Get to Google checking Internet connectivity
Date: 08/24-13:13:01.713666
SOURCE: 129.132.211.21x:50086
DEST: 173.194.112.210.80
22
./analyzerdynamic2.sh 129.132.211.21x 20130824 1320
Enter Comment. End it with ^D
EVENT: ET TROJAN Zeus Bot Get to Google checking Internet connectivity DATE: 08/24-
13:13:01.713666 SOURCE: 129.132.211.21x:50086 DEST: 173.194.112.210:80
Done. Comment stored in -rw-r--r-- 1 hall nsg 124 Aug 28 15:41 comment
DEBUG: starthour:201308241220 endhour:201308241320 startday:201308231320
port:-1
First DYNAMIC STAGE *************************************
nfdump -M /nfsen -R nfcapd.201308241220:nfcapd.201308241320 '( host
129.132.211.215 )' | grep ^2013 | awk '{ print $7 }' | sed s/:/ /g | awk '{ print $1 }' | grep -v
"129.132.211.21x:" | sort -u > ./analyzer_129.132.211.21x.outp Debug dnumber:90 Second
DYNAMIC STAGE *************************************
108.160.162.111
133 108.160.162.111 (89)
108.160.162.99
118 108.160.162.99 (88)
111.111.111.111
19 111.111.111.111 (87)
12.161.242.20
...
23
List Result
24
…
19 20 66.70.34.97 3 4.531722054 5.0000000000
19 19 111.111.111.111 148 .267413497 5.2631578947
14 15 77.72.169.160 24 4.633204633 6.6666666666
14 15 62.146.42.159 15 15.463917525 6.6666666666
14 15 98.139.205.30 9 20.930232558 6.6666666666
9 10 50.17.207.124 3 23.076923076 10.0000000000
6 7 98.139.114.30 6 31.578947368 14.2857142857
5 6 204.13.161.111 154 15.247524752 16.6666666666
5 6 54.225.249.200 14 28.000000000 16.6666666666
2 3 77.72.174.136 15 93.750000000 33.3333333333
1 2 140.116.72.75 2 100.000000000 50.0000000000
1 2 173.194.112.210 4 100.000000000 50.0000000000
1 2 66.196.120.57 2 100.000000000 50.0000000000
1 2 66.196.121.20 24 100.000000000 50.0000000000
Numberofips
Numberofips+
investigateddest
ip
Investigated
destip
Numberof
flows
Flowsin%
(1/Numberof
ipsincluding
destip)*100
Blacklist Check Routine
…
66.70.34.97
111.111.111.111
77.72.169.160
62.146.42.159
98.139.205.30
50.17.207.124
pbl:
!50.17.207.124
98.139.114.30
204.13.161.111
54.225.249.200
77.72.174.136
140.116.72.75
sbl hit: 140.116.72.75
173.194.112.210
66.196.120.57
66.196.121.20 25
Black List SBL Reference
http://www.spamhaus.org/sbl/query/SBL193024
Ref: SBL193024
140.116.72.75/32 is listed on the Spamhaus Block List - SBL
140.116.72.75/32 is listed on the Spamhaus Botnet C&C List - BGPCC
2013-08-26 15:56:50 GMT | edu.tw
Citadel botnet controller @140.116.72.75
Update Aug 26, 2013
Problem still exists, Citadel botnet controller located here:
http://dashuxmaecrme.com/wel/file.php
http://dashuxmaecrme.com/wel/qwrt.php
http://frontrunnings.com/fdet/file.php
http://joyrideengend.net/wel/file.php
http://spottingculde.com/wel/file.php
http://eenyellowredpf.su/wel/file.php
http://stabilitymess.net/wel/file.php
http://systemlevelge.com/wel/file.php
…
26
Possible to do’s
• Include (dest) Port in the analysis
• Automatically track compromised Ips
• Automatically analyse compromised Ips
• Automatically build and update CC lists
• Automatically correlation check between CC-
clusters and malware
27
Q&A
28
END
29
Christian Hallqvist / Network Security / ICT-Networks
hall@id.ethz.ch

Mais conteúdo relacionado

Semelhante a ETH_Anomaly-Detection-in-Netflows-PPT_v2_of_year_2013

Capture and replay hardware behaviour for regression testing and bug reporting
Capture and replay hardware behaviour for regression testing and bug reportingCapture and replay hardware behaviour for regression testing and bug reporting
Capture and replay hardware behaviour for regression testing and bug reporting
martin-pitt
 
A Framework for Protecting Worker Location Privacy in Spatial Crowdsourcing
A Framework for Protecting Worker Location Privacy in Spatial CrowdsourcingA Framework for Protecting Worker Location Privacy in Spatial Crowdsourcing
A Framework for Protecting Worker Location Privacy in Spatial Crowdsourcing
University of Southern California
 
DepokCyberSecurity - ServerHack - Wisolusindo -
DepokCyberSecurity - ServerHack - Wisolusindo -DepokCyberSecurity - ServerHack - Wisolusindo -
DepokCyberSecurity - ServerHack - Wisolusindo -
Adul Andreas
 
BRKDCT-3144 - Advanced - Troubleshooting Cisco Nexus 7000 Series Switches (20...
BRKDCT-3144 - Advanced - Troubleshooting Cisco Nexus 7000 Series Switches (20...BRKDCT-3144 - Advanced - Troubleshooting Cisco Nexus 7000 Series Switches (20...
BRKDCT-3144 - Advanced - Troubleshooting Cisco Nexus 7000 Series Switches (20...
aaajjj4
 
Tips on how to improve the performance of your custom modules for high volume...
Tips on how to improve the performance of your custom modules for high volume...Tips on how to improve the performance of your custom modules for high volume...
Tips on how to improve the performance of your custom modules for high volume...
Odoo
 

Semelhante a ETH_Anomaly-Detection-in-Netflows-PPT_v2_of_year_2013 (20)

OpenStack - A Python-based Cloud-Software
OpenStack - A Python-based Cloud-SoftwareOpenStack - A Python-based Cloud-Software
OpenStack - A Python-based Cloud-Software
 
Metrics with Ganglia
Metrics with GangliaMetrics with Ganglia
Metrics with Ganglia
 
Capture and replay hardware behaviour for regression testing and bug reporting
Capture and replay hardware behaviour for regression testing and bug reportingCapture and replay hardware behaviour for regression testing and bug reporting
Capture and replay hardware behaviour for regression testing and bug reporting
 
test
testtest
test
 
LISA2019 Linux Systems Performance
LISA2019 Linux Systems PerformanceLISA2019 Linux Systems Performance
LISA2019 Linux Systems Performance
 
Best Practices in Handling Performance Issues
Best Practices in Handling Performance IssuesBest Practices in Handling Performance Issues
Best Practices in Handling Performance Issues
 
firewall
firewallfirewall
firewall
 
The Data Center and Hadoop
The Data Center and HadoopThe Data Center and Hadoop
The Data Center and Hadoop
 
Trash Robotic Router Platform - David Melendez - Codemotion Rome 2015
Trash Robotic Router Platform - David Melendez - Codemotion Rome 2015Trash Robotic Router Platform - David Melendez - Codemotion Rome 2015
Trash Robotic Router Platform - David Melendez - Codemotion Rome 2015
 
A Framework for Protecting Worker Location Privacy in Spatial Crowdsourcing
A Framework for Protecting Worker Location Privacy in Spatial CrowdsourcingA Framework for Protecting Worker Location Privacy in Spatial Crowdsourcing
A Framework for Protecting Worker Location Privacy in Spatial Crowdsourcing
 
Linux 系統管理與安全:進階系統管理系統防駭與資訊安全
Linux 系統管理與安全:進階系統管理系統防駭與資訊安全Linux 系統管理與安全:進階系統管理系統防駭與資訊安全
Linux 系統管理與安全:進階系統管理系統防駭與資訊安全
 
DepokCyberSecurity - ServerHack - Wisolusindo -
DepokCyberSecurity - ServerHack - Wisolusindo -DepokCyberSecurity - ServerHack - Wisolusindo -
DepokCyberSecurity - ServerHack - Wisolusindo -
 
Tool sdl2pml
Tool sdl2pmlTool sdl2pml
Tool sdl2pml
 
Open daylight and Openstack
Open daylight and OpenstackOpen daylight and Openstack
Open daylight and Openstack
 
BRKDCT-3144 - Advanced - Troubleshooting Cisco Nexus 7000 Series Switches (20...
BRKDCT-3144 - Advanced - Troubleshooting Cisco Nexus 7000 Series Switches (20...BRKDCT-3144 - Advanced - Troubleshooting Cisco Nexus 7000 Series Switches (20...
BRKDCT-3144 - Advanced - Troubleshooting Cisco Nexus 7000 Series Switches (20...
 
Combo fix
Combo fixCombo fix
Combo fix
 
Tips on how to improve the performance of your custom modules for high volume...
Tips on how to improve the performance of your custom modules for high volume...Tips on how to improve the performance of your custom modules for high volume...
Tips on how to improve the performance of your custom modules for high volume...
 
Ns network simulator
Ns network simulatorNs network simulator
Ns network simulator
 
Combo fix
Combo fixCombo fix
Combo fix
 
Combo fix
Combo fixCombo fix
Combo fix
 

ETH_Anomaly-Detection-in-Netflows-PPT_v2_of_year_2013

  • 1. © ETH Zürich | ICT-Networks/NSG christian.hallqvist@id.ethz.ch 27.08.2015 Simple Anomaly Detection via Netflows
  • 2. 2 The Big Question: What Is Under The Radar?
  • 3. First Goal (1) To create an automated method of detecting unusual connections and/or anomalies with netflows, thereby finding compromised hosts 3
  • 4. Top Down Method Works Well With IDS 4 N=100 P2P BitTorrent transfer „Deviation“ (total traffic)
  • 5. Netflow Anomaly Detection Top Down Strategy Analyzing the total traffic down to individual hosts by detecting behavioral deviations. Been there and done that with netflows (2008). The problem: Even though a malicious traffic event is usually an anomaly, an anomaly is not always a malicious event 5
  • 6. With an automated method, finding the correlating netflows of incidents regardless of source of information (IDS, Switch, AV, User, Admin, Netflows) New Goal (2) 6
  • 7. Netflow Anomaly Detection Bottom up Strategy Viewing what an individual host is doing compared to the general population 7
  • 8. Common Problems Regardless of Source of Information • Vague Indications • Detected Anomalies • Recurring Compromise • False Negatives • False Positives 8
  • 12. Pros and Cons With This Method Pro: It is automatic and indeed sometimes successfull Con: It may take a long time to run Again: All anomalies are not malicious. The potential problem is when the individual host is generating harmless but very diverse unusal traffic. Both pros and cons: It is possible to automatically sort the connections based on how usual/unusal they are 12
  • 14. A Real Example (Switch) [SWITCH-CERT #22814x ] Most likely compromised system [129.132.208.10x] [Botnet] Based on received information about a ‘malicious IRC command master at 183.203.15.205 2013-08-14 17:40:08.070 14
  • 15. ./analyzerdynamic2.sh 129.132.208.10x 20130814 1800 Enter Comment. End it with ^D Subject: [SWITCH-CERT #228144] Most likely compromised system [129.132.208.10x ...] [Botnet] Done. Comment stored in -rw-r--r-- 1 hall nsg 93 Aug 22 18:05 comment DEBUG: starthour:201308141700 endhour:201308141800 startday:201308131800 port:-1 First DYNAMIC STAGE ************************************* nfdump -M /nfsen -R nfcapd.201308141700:nfcapd.201308141800 '( host 129.132.208.10x )' | grep ^2013 | awk '{ print $7 }' | sed s/:/ /g | awk '{ print $1 }' | grep -v "129.132.208.10x:" | sort -u > ./analyzer_129.132.208.10x.outp Debug dnumber:41 Second DYNAMIC STAGE ************************************* 107.21.234.205 112 107.21.234.205 (40) 108.160.162.53 1938 108.160.162.53 (39) 108.160.163.46 229 108.160.163.46 (38) 12.130.131.80 120 12.130.131.80 (37) ... 15
  • 16. … 120 121 12.130.131.80 10 5.0000000 .8264462 112 113 107.21.234.205 11 4.5454545 .8849557 95 96 77.67.22.188 2 .7142857 1.0416666 45 46 62.146.92.202 12 1.090909 2.17391304 34 35 137.205.124.72 3 .2497918 2.8571428 29 30 54.224.40.139 10 4.8309178 3.3333333 27 28 134.60.1.5 13 .0336482 3.5714285 23 24 88.156.222.90 4 1.8691588 4.1666666 17 18 178.63.20.18 5 2.6041666 5.5555555 7 8 193.36.36.16 91 5.6627255 12.5000000 3 3 183.203.8.238 1 .0959692 33.3333333 2 3 183.203.15.205 34 3.333333 33.333333316 List Result Numberofips Numberofips+ investigateddest ip Investigated destip Numberof flows Flowsin% (1/Numberof ips+ investigated destip)*100
  • 19. The Concept in a Nutshell 1. Find all connections around the investigated IP over a 60 minute period 2. Take those connections and rate how usual (or unusal) these are in the general population over a 24hr period 19
  • 20. Find All Connections Around the Investigated IP Over a 60 Minute Period 20 Time point of interest for the investigated ip Internet Investigated IP at ETH network
  • 21. Take Those Connections and Rate How Usual (or Unusal) These are in the General Population Over a 24hr Period InternetETH network
  • 22. Another Real Example (IDS) IDS Event with destination google: EVENT: ET TROJAN Zeus Bot Get to Google checking Internet connectivity Date: 08/24-13:13:01.713666 SOURCE: 129.132.211.21x:50086 DEST: 173.194.112.210.80 22
  • 23. ./analyzerdynamic2.sh 129.132.211.21x 20130824 1320 Enter Comment. End it with ^D EVENT: ET TROJAN Zeus Bot Get to Google checking Internet connectivity DATE: 08/24- 13:13:01.713666 SOURCE: 129.132.211.21x:50086 DEST: 173.194.112.210:80 Done. Comment stored in -rw-r--r-- 1 hall nsg 124 Aug 28 15:41 comment DEBUG: starthour:201308241220 endhour:201308241320 startday:201308231320 port:-1 First DYNAMIC STAGE ************************************* nfdump -M /nfsen -R nfcapd.201308241220:nfcapd.201308241320 '( host 129.132.211.215 )' | grep ^2013 | awk '{ print $7 }' | sed s/:/ /g | awk '{ print $1 }' | grep -v "129.132.211.21x:" | sort -u > ./analyzer_129.132.211.21x.outp Debug dnumber:90 Second DYNAMIC STAGE ************************************* 108.160.162.111 133 108.160.162.111 (89) 108.160.162.99 118 108.160.162.99 (88) 111.111.111.111 19 111.111.111.111 (87) 12.161.242.20 ... 23
  • 24. List Result 24 … 19 20 66.70.34.97 3 4.531722054 5.0000000000 19 19 111.111.111.111 148 .267413497 5.2631578947 14 15 77.72.169.160 24 4.633204633 6.6666666666 14 15 62.146.42.159 15 15.463917525 6.6666666666 14 15 98.139.205.30 9 20.930232558 6.6666666666 9 10 50.17.207.124 3 23.076923076 10.0000000000 6 7 98.139.114.30 6 31.578947368 14.2857142857 5 6 204.13.161.111 154 15.247524752 16.6666666666 5 6 54.225.249.200 14 28.000000000 16.6666666666 2 3 77.72.174.136 15 93.750000000 33.3333333333 1 2 140.116.72.75 2 100.000000000 50.0000000000 1 2 173.194.112.210 4 100.000000000 50.0000000000 1 2 66.196.120.57 2 100.000000000 50.0000000000 1 2 66.196.121.20 24 100.000000000 50.0000000000 Numberofips Numberofips+ investigateddest ip Investigated destip Numberof flows Flowsin% (1/Numberof ipsincluding destip)*100
  • 26. Black List SBL Reference http://www.spamhaus.org/sbl/query/SBL193024 Ref: SBL193024 140.116.72.75/32 is listed on the Spamhaus Block List - SBL 140.116.72.75/32 is listed on the Spamhaus Botnet C&C List - BGPCC 2013-08-26 15:56:50 GMT | edu.tw Citadel botnet controller @140.116.72.75 Update Aug 26, 2013 Problem still exists, Citadel botnet controller located here: http://dashuxmaecrme.com/wel/file.php http://dashuxmaecrme.com/wel/qwrt.php http://frontrunnings.com/fdet/file.php http://joyrideengend.net/wel/file.php http://spottingculde.com/wel/file.php http://eenyellowredpf.su/wel/file.php http://stabilitymess.net/wel/file.php http://systemlevelge.com/wel/file.php … 26
  • 27. Possible to do’s • Include (dest) Port in the analysis • Automatically track compromised Ips • Automatically analyse compromised Ips • Automatically build and update CC lists • Automatically correlation check between CC- clusters and malware 27
  • 29. END 29 Christian Hallqvist / Network Security / ICT-Networks hall@id.ethz.ch