O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
l/ /XPRA

Anomaly Detection

How to Find What You Didn't
Know to Look For

Practical Machine Learning Webinar with Ted Dun...
Anomaly Detection: 
How To Find What You Didn't Know to Look For

Ted Dunning,  Chief Applications Architect MapR Technolo...
A New Look at Anomaly Detection
by Ted Dunning and Ellen Friedman © June 2014 (published by O'Reilly)

WT“ '—a—, —_ ! 

‘‘...
Practical Machine Learning series (O’Reilly)

0

I((/ 

Machine learning is becoming mainstream
Need pragmatic approaches ...
Anomaly Detection

oaoim-uni-a--nun MAPR I
Who Needs Anomaly Detection? 

 ‘~ .  “"—‘*“: 
l i ‘    ’ J
.1 T‘ E '|  ; _ - _‘ . ..—"
Q 5..  _  ‘ Il  I i.  i ; 

H , - ...
Who Needs Anomaly Detection? 

    
  

I
‘I L
1

Feedback from

‘ manufacturing assembly

lines

N/ PR
Who Needs Anomaly Detection? 

 

CA

Monitoring data traffic on
communication networks

M/ NPR >—
What is Anomaly Detection? 

- The goal is to discover rare events
— especially those that shouldn't have happened

- Find...
Spot the Anomaly

Signal
10 15 20

0 500

(‘A

1000

1500

2000

2500

3000

M/ NPR
Spot the Anomaly

Signal
10 15 20

ICA‘

1000

 

1500

Looks pretty
anomalous

to me

2000

2500

 

3000

MAPR ‘‘
Spot the Anomaly

15

Signal
10

ICA‘

Will the real anomaly
please stand up? 

1000 1500 2000

2500

3000

M/ NPR‘
Find

Basic idea: 
"normal" first

oaouuuuuuav-own NAPR I3
Steps in Anomaly Detection

- Build a model:  Collect and process data for training a model

- Use the machine learning mo...
How hard is it to set an alert for anomalies? 

-20246810

Grey data is from normal events;  x's are anomalies. 
Where wou...
Basic idea: 
Set adaptive thresholds

oaoiau-uni-an-no-o NVPR Ia
What Are We Really Doing

- We want action when something breaks
(dies/ falls over/ otherwise gets in trouble)

- But acti...
A Second Look

Signal
10 15 20

(‘A

500

1000

1500

2000

2500

3000

M/ NPR it
A Second Look

Signal
10 15 20

99.9%-ile

500 I 1000 1 500 2000 2500 3000

1

Lilollflmfllothlialovol  I9
New algorithm:  t-digest

oaoiau-uni-an-w-o NVPR zo
How Hard Can it Be? 

  

V I
l —'lill| n

l -‘1IIiilnl. IitIi 99.90/°_i| e

*7” Alarm l

NfPR.
Detecting Anomalies in Sporadic Events

 

3
3
S
9
00 0.0 0.4 0.0 00 1.0
FDVNKOIIIOWIOIOVIOIIUUOIII

 ozcimunvomusoooo Ni/...
Using t-Digest

- Apache Mahout uses t—digest as an on-line percentile estimator
— very high accuracy for extreme tails
- ...
Already Done?  Etsy Skyline? 

WEI-

uuiuirown pnuon. Iimn. n-u. nIi«'p lo “'

ll’ ll. ’ ' V ‘ "
 . 
in .1.
——

9.. .‘. .....
What About This? 

-20246810

(‘A‘

10

15

~ MAPR
Model Delta Anomaly Detection

   

' ~‘lIiuiiii. Iii4.i gg4g%_,1L. 

 

—? * Alarm ! 

N/ PR
The Real Inside Scoop

- The model-delta anomaly detector is really just a sum of random
variables

— the model we know ab...
Anomalies among sporadic events

oaoiau-in-an-w-o NVPR 20
Sporadic Web Traffic to an e-Business Site

It's important to know if traffic is stopped or
delayed because of a problem. ...
Sporadic Web Traffic to an e-Business Site

It's important to know if traffic is stopped or
delayed because of a problem. ...
C? 

Basic idea: 

Time interval between events is how

to convert to something useful you
can measure

ozoim-I I-a-can NV...
Sporadic Events:  Finding Normal and Anomalous Patterns

Time between intervals is much more usable than absolute times

C...
Event Stream (timing)

(‘A

Events of various types arrive at irregular intervals
— we can assume Poisson distribution

Th...
Converting Event Times to Anomaly

l 99.9%-iie| 

 L ‘ V. ‘ ‘J H '|  I Ill‘
2 lllill. iill. lllllllllll'lllll'illllllll. ‘...
But in the real world,  event
rates often change

oaoim-int-a-new-o NVPR an
lime Intervals Are Key to Modeling Sporadic Events

1

interval
0.2 0.4 0.6 0.8

 

0.00 0.05 0.10 0.15 0.20
0

0.0 0.5 1....
Model-Scaled Intervals Solve the Problem

M
A

Scaled Imerval

‘N
1-

Q
o

0.4

0.0

0.0

0.5

1.0

1.5

Time (days)

2.0
...
Detecting Anomalies in Sporadic Events

 i @

predictor

Incoming
events

 

    

N/ NPR
Detecting Anomalies in Sporadic Events

 i @
A

predictor

N. Ili1
hv. li)7y

Incoming
events

     

N/ NPR
Slipped Week:  Simple Rate Predictor

MelnPe9e'l'ra1l‘lc
§
5 r
f §
3 A B C 0

Nov 0? Nov 07 Nov 12 Nov I7 Nov 2’?  Nov 2? ...
Poisson Distribution

- Time between events is exponentially distributed

At ~ 2.6”
- This means that long delays are expo...
Seasonality Poses a Challenge

Christmas Trafllc

 

E
1 till 
~ .   .  Ill
. ~r‘ iN‘1.'*1.'1,~il ‘i Ni (ii 9}‘ ‘ Vi" ‘ N ...
Something more is needed 

Christmas Tmfllc

N‘. SlI000

(‘A

 

MN'-‘R : ~
Sometimes need a better rate predictor. ..

 i @
A

predictor

N. Ili1
hv. li)7y

Incoming
events

     

N/ NPR 11
A New Rate Predictor for Sporadic Events

y :1 in no no is
Omen Iran Nun than leneue tenure
Dfiflfil '40 V ifl U9 lfl 3
2000-11...
Improved Prediction with Adaptive Modeling

clrleeneehedlaten

been Decle Dealt Dee Deena Dean Deal! 
on-

FM O20l4Meo0(1e...
Improving Security with Anomaly Detection

There are a variety of ways in which

businesses are at risk of attack

Anomaly...
No Phishing Allowed! 

INPR1
(‘A

 

.4 _—. ._. .—. —_—, ..-. L-.  ,
The Attack: 

 

A-(/ “

1 M/ NPR an
The Attack: 

% imp:  .'. ~,ouberli. t1« com

(‘A‘

 

1 M/ NPR '-1
The Attack: 

% imp:  .'. ~,outi-InIi. tl~ com

(‘A‘

 

Fraudlsite , 

 
    

 

l Real Site

1 M/ NPR
The Attack: 

I Fraud Site

l

l Real Site l

1. I
I

NUXPR
The Attack: 

% imp:  .'. ~,outi-InIi. tl~ com

(‘A‘

 

Fraudlsite , 

 
   
 

 

Real Site

1 M/ NPR
Force phishers to include images

  
  
     

  

Dynamic security elements
must be interpreted by
user

(‘A‘

Characteri...
Normal Event Flow

| mage—2
j

Kfli’

§

login

fl

ozouwou Yum-otoon  56
Phishing Flow

Q

l

(‘A

Real login

é  "
3 _, 

Fake login

MN"-’R
Web logs capture events

% Imp:  .'. ~,ououl: .t1« com

Fraudlsite

(‘A‘

 

Real Site

‘ M/ NPR
Web logs capture events

% Imps . '.~, oubulw1« com

Fraud Site

1

K3;

  

 

Real Site

- M/ NPR
Kk‘

Normal Pattern

Images
II II I Lofiin

~ M/ NPR m
IP1

IPZ

Phishing Pattern is Anomalous

Ima es
II II I

A D

Fraud login

_______LLJJ_J_Tr_rF_F_______________r_____L___—...
Key Observations

- Regardless of exact details,  there are observable patterns

— Event stream per user shows these patte...
Keep Ahead of the Bad Guys

cg’

Fraudsters constantly change their tricky ways

You must look for new patterns without kn...
Anomaly Detection + Classification 9 Useful Pair

- Use the AD model to detect anomalies in new data
— Methods such as clu...
Recap (out of order)

Anomaly detection is best done with a probability model
-log p is a good way to convert to anomaly m...
Recap

- Different systems require different models

- Continuous time-series
— sparse coding to build signal model

- Eve...
Why Use Anomaly Detection?
Keep in mind. ..

:2

Model normal,  then find
anomalies

-IOIIOIVO

t-digest for adaptive threshold

Probabilistic models...
Keep in mind. ..

- 1'Ime intervals are key for
sporadic events

- Complex time shift to predict
rate with seasonality

- ...
A New Look at Anomaly Detection
by Ted Dunning and Ellen Friedman © June 2014 (published by O'Reilly)

WT“ '—a—. —_ ! 

‘‘...
Coming in October:  Time Series Databases
by Ted Dunning and Ellen Friedman © Oct 2014 (published by O‘Reilly)

 

Fig O20...
Thank you for coming today! 

oaouu-an-emu-an MAPR 12
.  §
‘

‘
‘
,(‘A

Sandbox

M/ NPR
Próximos SlideShares
Carregando em…5
×

Anomaly Detection: When You Don't Know What You Need to Find

4.387 visualizações

Publicada em

In a complex world of rapidly changing information, it’s not easy to find something that is rare – but it’s even harder if you do not know what to look for. In this webinar, Ted Dunning and Ellen Friedman, authors of the O’Reilly ebook, Practical Machine Learning: A New Look at Anomaly Detection, discussed novel ways to build anomaly detectors that can handle various types of large scale data, in simple systems and in complex ones, as well.

Publicada em: Tecnologia
  • Seja o primeiro a comentar

Anomaly Detection: When You Don't Know What You Need to Find

  1. 1. l/ /XPRA Anomaly Detection How to Find What You Didn't Know to Look For Practical Machine Learning Webinar with Ted Dunning and Ellen Friedman :2:
  2. 2. Anomaly Detection: How To Find What You Didn't Know to Look For Ted Dunning, Chief Applications Architect MapR Technologies Email tdunning@mapr. com tdunning@agache. org Twitter @Ted_Dunning Ellen Friedman, Consultant and Commentator Email e| lenf@apache. org Twitter @E| |en_Friedman Follow the conversation on twitter: #ADWebinar KR’ Ni/ NPR 2
  3. 3. A New Look at Anomaly Detection by Ted Dunning and Ellen Friedman © June 2014 (published by O'Reilly) WT“ '—a—, —_ ! ‘‘':1 J v . -= ~’= ll= <n Uh‘ ” " e-book available courtesy of MapR ' " ht_tg: _// bitly/1lQ9QuL arffi, -.-0 ) / / ((/ i M/ NPR ~
  4. 4. Practical Machine Learning series (O’Reilly) 0 I((/ Machine learning is becoming mainstream Need pragmatic approaches that take into account real world business settings: — Time to value — Limited resources - Availability of data — Expertise and cost of team to develop and to maintain system Look for approaches with big benefits for the effort expended tr: / ,. cg" . ‘ M/ NPR l
  5. 5. Anomaly Detection oaoim-uni-a--nun MAPR I
  6. 6. Who Needs Anomaly Detection? ‘~ . “"—‘*“: l i ‘ ’ J .1 T‘ E '| ; _ - _‘ . ..—" Q 5.. _ ‘ Il I i. i ; H , - . 5. i ‘- vb, . P. ‘ ‘°F; ’-$l"}' i ll " i ' - l , T ' A - ‘, i w in :1 3 L §, .., « En iv fi H. ‘)5’; l 2 . ‘ . hi it ‘Ii (3 ( “o)’. r’ fl‘) X A ( i 7; A5, 1 I. i/ /1. . .’V / (‘A Utility providers using smart meters M/ PRi
  7. 7. Who Needs Anomaly Detection? I ‘I L 1 Feedback from ‘ manufacturing assembly lines N/ PR
  8. 8. Who Needs Anomaly Detection? CA Monitoring data traffic on communication networks M/ NPR >—
  9. 9. What is Anomaly Detection? - The goal is to discover rare events — especially those that shouldn't have happened - Find a problem before other people see it — especially before it causes a problem for customers - Why is this a challenge? — I don't know what an anomaly looks like (yet) £2" _~ in Mn; O( lpm. .,. .-; ,u Ni/ NPR
  10. 10. Spot the Anomaly Signal 10 15 20 0 500 (‘A 1000 1500 2000 2500 3000 M/ NPR
  11. 11. Spot the Anomaly Signal 10 15 20 ICA‘ 1000 1500 Looks pretty anomalous to me 2000 2500 3000 MAPR ‘‘
  12. 12. Spot the Anomaly 15 Signal 10 ICA‘ Will the real anomaly please stand up? 1000 1500 2000 2500 3000 M/ NPR‘
  13. 13. Find Basic idea: "normal" first oaouuuuuuav-own NAPR I3
  14. 14. Steps in Anomaly Detection - Build a model: Collect and process data for training a model - Use the machine learning model to determine what is the normal pattern - Decide how far away from this normal pattern you’ll consider to be anomalous - Use the AD model to detect anomalies in new data — Methods such as clustering for discovery can be helpful (‘A 1 M/ NPR it
  15. 15. How hard is it to set an alert for anomalies? -20246810 Grey data is from normal events; x's are anomalies. Where would you set the threshold? (‘A M/ PRi
  16. 16. Basic idea: Set adaptive thresholds oaoiau-uni-an-no-o NVPR Ia
  17. 17. What Are We Really Doing - We want action when something breaks (dies/ falls over/ otherwise gets in trouble) - But action is expensive - So we don't want too many false alarms - And we don't want too many false negatives - What’s the right threshold to set for alerts? — We need to trade off costs (‘A M/ NPR i
  18. 18. A Second Look Signal 10 15 20 (‘A 500 1000 1500 2000 2500 3000 M/ NPR it
  19. 19. A Second Look Signal 10 15 20 99.9%-ile 500 I 1000 1 500 2000 2500 3000 1 Lilollflmfllothlialovol I9
  20. 20. New algorithm: t-digest oaoiau-uni-an-w-o NVPR zo
  21. 21. How Hard Can it Be? V I l —'lill| n l -‘1IIiilnl. IitIi 99.90/°_i| e *7” Alarm l NfPR.
  22. 22. Detecting Anomalies in Sporadic Events 3 3 S 9 00 0.0 0.4 0.0 00 1.0 FDVNKOIIIOWIOIOVIOIIUUOIII ozcimunvomusoooo Ni/ NPR 22
  23. 23. Using t-Digest - Apache Mahout uses t—digest as an on-line percentile estimator — very high accuracy for extreme tails - new in version Mahout v 0.9 - t—digest also available elsewhere — in streamlib (open source library on github) - standalone (github and Maven Central) - What's the big deal with anomaly detection? - This looks like a solved problem , (‘, {' _. 0.1404 A/ wan 23
  24. 24. Already Done? Etsy Skyline? WEI- uuiuirown pnuon. Iimn. n-u. nIi«'p lo “' ll’ ll. ’ ' V ‘ " . in .1. —— 9.. .‘. .. mi. .. MAPR . :
  25. 25. What About This? -20246810 (‘A‘ 10 15 ~ MAPR
  26. 26. Model Delta Anomaly Detection ' ~‘lIiuiiii. Iii4.i gg4g%_,1L. —? * Alarm ! N/ PR
  27. 27. The Real Inside Scoop - The model-delta anomaly detector is really just a sum of random variables — the model we know about already — and a normally distributed error - The output (delta) is (roughly) the log probability of the sum distribution (really 62) - Thinking about probability distributions is good - But how do you handle AD in systems with sporadic events? , (‘, {' _. 0.1404 A/ wan 27
  28. 28. Anomalies among sporadic events oaoiau-in-an-w-o NVPR 20
  29. 29. Sporadic Web Traffic to an e-Business Site It's important to know if traffic is stopped or delayed because of a problem. .. But visits to site normally come at varying intervals. How long after the last event should you begin to worry?
  30. 30. Sporadic Web Traffic to an e-Business Site It's important to know if traffic is stopped or delayed because of a problem. .. But visits to site normally come at varying intervals. - 7- - ‘ " And how do you let your CEO sleep through the night?
  31. 31. C? Basic idea: Time interval between events is how to convert to something useful you can measure ozoim-I I-a-can NVPR. at
  32. 32. Sporadic Events: Finding Normal and Anomalous Patterns Time between intervals is much more usable than absolute times Counts don't link as directly to probability models Time interval is log p This is a big deal (‘A it M/ NPR
  33. 33. Event Stream (timing) (‘A Events of various types arrive at irregular intervals — we can assume Poisson distribution The key question is whether frequency has changed relative to expected values — This shows up as a change in interval Want alert as soon as possible M/ NPR -. ~
  34. 34. Converting Event Times to Anomaly l 99.9%-iie| L ‘ V. ‘ ‘J H '| I Ill‘ 2 lllill. iill. lllllllllll'lllll'illllllll. ‘liilillillllli'l'lll»l. 'llli~l'lililll. .llil ill’ M/ PR=
  35. 35. But in the real world, event rates often change oaoim-int-a-new-o NVPR an
  36. 36. lime Intervals Are Key to Modeling Sporadic Events 1 interval 0.2 0.4 0.6 0.8 0.00 0.05 0.10 0.15 0.20 0 0.0 0.5 1.0 1.5 2.0 2.5 3.0 (‘A 1 M/ NPR 1
  37. 37. Model-Scaled Intervals Solve the Problem M A Scaled Imerval ‘N 1- Q o 0.4 0.0 0.0 0.5 1.0 1.5 Time (days) 2.0 2.5 3.0 Clflllflmflhthlitiopci 37
  38. 38. Detecting Anomalies in Sporadic Events i @ predictor Incoming events N/ NPR
  39. 39. Detecting Anomalies in Sporadic Events i @ A predictor N. Ili1 hv. li)7y Incoming events N/ NPR
  40. 40. Slipped Week: Simple Rate Predictor MelnPe9e'l'ra1l‘lc § 5 r f § 3 A B C 0 Nov 0? Nov 07 Nov 12 Nov I7 Nov 2’? Nov 2? Dvc 02 Dairy (‘A M/ NPR:
  41. 41. Poisson Distribution - Time between events is exponentially distributed At ~ 2.6” - This means that long delays are exponentially rare P(At > T) = e‘” - log P(At > T) = AT - If we know 2. we can select a good threshold — or we can pick a threshold empirically (‘A“ _- 1.. .., ..- Nl/ NPR -11
  42. 42. Seasonality Poses a Challenge Christmas Trafllc E 1 till ~ . . Ill . ~r‘ iN‘1.'*1.'1,~il ‘i Ni (ii 9}‘ ‘ Vi" ‘ N ‘W ' ' A NCNI7 Nov27 0-1207 0 :17 mi. (‘A MAPR : .
  43. 43. Something more is needed Christmas Tmfllc N‘. SlI000 (‘A MN'-‘R : ~
  44. 44. Sometimes need a better rate predictor. .. i @ A predictor N. Ili1 hv. li)7y Incoming events N/ NPR 11
  45. 45. A New Rate Predictor for Sporadic Events y :1 in no no is Omen Iran Nun than leneue tenure Dfiflfil '40 V ifl U9 lfl 3 2000-11-23 13100100 601 030 I91 445 610 545 2000-11-23 14:00:00 755 601 630 091 705 010 2000-11-23 15:00:00 087 755 601 630 607 2000-11-23 10:00:00 960 007 755 001 002 :2; Delete: x) A c120i4moNi«~iao9u 45
  46. 46. Improved Prediction with Adaptive Modeling clrleeneehedlaten been Decle Dealt Dee Deena Dean Deal! on- FM O20l4Meo0(1e(Nio0o90| 46
  47. 47. Improving Security with Anomaly Detection There are a variety of ways in which businesses are at risk of attack Anomaly detection is a powerful / :55‘ $1 ‘ tool for reducing this risk ~ _. ‘I’ 5"" 3' 3%" “ . ‘V '. “ 5‘ Here's how £5,
  48. 48. No Phishing Allowed! INPR1
  49. 49. (‘A .4 _—. ._. .—. —_—, ..-. L-. ,
  50. 50. The Attack: A-(/ “ 1 M/ NPR an
  51. 51. The Attack: % imp: .'. ~,ouberli. t1« com (‘A‘ 1 M/ NPR '-1
  52. 52. The Attack: % imp: .'. ~,outi-InIi. tl~ com (‘A‘ Fraudlsite , l Real Site 1 M/ NPR
  53. 53. The Attack: I Fraud Site l l Real Site l 1. I I NUXPR
  54. 54. The Attack: % imp: .'. ~,outi-InIi. tl~ com (‘A‘ Fraudlsite , Real Site 1 M/ NPR
  55. 55. Force phishers to include images Dynamic security elements must be interpreted by user (‘A‘ Characteristics of image element - can be passed on by login form ,1 1.. ... . . . A/ VKPR 5,5,
  56. 56. Normal Event Flow | mage—2 j Kfli’ § login fl ozouwou Yum-otoon 56
  57. 57. Phishing Flow Q l (‘A Real login é " 3 _, Fake login MN"-’R
  58. 58. Web logs capture events % Imp: .'. ~,ououl: .t1« com Fraudlsite (‘A‘ Real Site ‘ M/ NPR
  59. 59. Web logs capture events % Imps . '.~, oubulw1« com Fraud Site 1 K3; Real Site - M/ NPR
  60. 60. Kk‘ Normal Pattern Images II II I Lofiin ~ M/ NPR m
  61. 61. IP1 IPZ Phishing Pattern is Anomalous Ima es II II I A D Fraud login _______LLJJ_J_Tr_rF_F_______________r_____L___—————T ‘I1_U_j[1 Real login Images B C MAPR
  62. 62. Key Observations - Regardless of exact details, there are observable patterns — Event stream per user shows these patterns — Phishing will have different patterns at much lower rate - Measuring statistical surprise gives a good anomaly indicator to find fraud or malfunctions , (‘, {" _. M A/ wan 5;
  63. 63. Keep Ahead of the Bad Guys cg’ Fraudsters constantly change their tricky ways You must look for new patterns without knowing what the anomalies will look like Use practical machine learning models for anomaly detection to find what you didn't know to look for . -im. ... .t. ... ,l, ... .,, . Nt/ NPR 53
  64. 64. Anomaly Detection + Classification 9 Useful Pair - Use the AD model to detect anomalies in new data — Methods such as clustering for discovery can be helpful - Once you have well-defined models in your system, you may also want to use classification to tag those - Continue to use the AD model to find new anomalies I((/ M/ NPR ll
  65. 65. Recap (out of order) Anomaly detection is best done with a probability model -log p is a good way to convert to anomaly measure Adaptive quantile estimation (t-digest) works for auto-setting thresholds I((/ M/ NPR
  66. 66. Recap - Different systems require different models - Continuous time-series — sparse coding to build signal model - Events in time - rate model base on variable rate Poisson — segregated rate model - Events with labels - language modeling - hidden Markov models cg’ _- , i4 u. .u r. .,m. ‘.. .,a-s M/ NPR
  67. 67. Why Use Anomaly Detection?
  68. 68. Keep in mind. .. :2 Model normal, then find anomalies -IOIIOIVO t-digest for adaptive threshold Probabilistic models for complex patterns : _*nlvl Mxk You M/ NPR an
  69. 69. Keep in mind. .. - 1'Ime intervals are key for sporadic events - Complex time shift to predict rate with seasonality - Sequence of events reveals phishing attack 0:’ 2 ' :53» I his nu I-I HI luv Iul & nzcmauu vumumu [Vt/ ‘PR so
  70. 70. A New Look at Anomaly Detection by Ted Dunning and Ellen Friedman © June 2014 (published by O'Reilly) WT“ '—a—. —_ ! ‘‘':1 J v . arm" Uh‘ ” " e-book available courtesy of MapR ' " ht_tQ: _// bit. ly/1iQ9QuL arffi, -.-0 ) / / ((/ I M/ NPR
  71. 71. Coming in October: Time Series Databases by Ted Dunning and Ellen Friedman © Oct 2014 (published by O‘Reilly) Fig O20llMaoflVoflInoIo9o| 11
  72. 72. Thank you for coming today! oaouu-an-emu-an MAPR 12
  73. 73. . § ‘ ‘ ‘
  74. 74. ,(‘A Sandbox M/ NPR

×