Designing Cybersecurity Policies with Field Experiments

Designing Cybersecurity
Policies with Field Experiments
Gene Moo Lee
University of Texas at Austin
Joint work with Shu He, John S. Quarterman, Andrew B. Whinston
Supported by NSF 1228990
February 25, 2015
KAIST

Gene Moo Lee, KAIST, Feb 2015
“Although the threats are serious and they
constantly evolve, I believe that if we address them
effectively, we can ensure that the Internet remains
an engine for economic growth and a platform for
the free exchange of ideas.”
—Barack Obama
2

Motivation
• Inadequate cybersecurity is a serious threat
• avg cost $3.5 million in 2013, 15% increase
• # of compromises increased by 25%
• data breaches of 2.6 million Target consumers
• U.S. government’s measures
• Cybersecurity Policy Review (2009)
• Executive Order 13636 (2013) “Improving Critical
Infrastructure Cybersecurity”
3

Approaches
• Technical approaches:
• spam filtering, intrusion detection systems (IDS), digital
forensics
• Sahami et al. (1998), Cormack and Lynam (2007), Denning
(1987), Lee and Stolfo (1998), Casey (2011), Taylor et al.
(2014)
• Economic approaches:
• underinvestment due to (1) information asymmetry, (2)
network externalities, (3) moral hazards
• van Eeten et al. (2011), Moore and Clayton (2011), Arora
et al. (2004), D’Arcy et al. (2009), Wood and Rowe (2011)
4

Our approach
• We found evidence that spam evaluation publication help improving
security levels in country level
• Quarterman et al. (2012), Qian et al. (2013)
• Use outbound spam to estimate latent security level
• 90% spam is from compromised computers controlled by botnets
(Rao and Reiley 2012, Moore and Clayton 2011)
• Ultimate goal:
• Evaluate the effectiveness in organizational level
• government sponsored institution to monitor and evaluate
organizational security levels (Moody’s, S&P for bonds)
• Counterfactual policy analysis with randomized field experiments
5

Research questions
1. Our goal is to set up an independent institution to evaluate
and monitor all organizations’ cybersecurity level
2. Does information disclosure change organizational
behaviour? In other words, spam reduce?
• Method: Randomized field experiment
• Two treatment groups with different info disclosure
• Two cycles of emails at January/March 2014
• A website built on Google cloud
6

Experimental design
• 7919 US organizations, three groups: control, private, public
• Private treatment: email with spam volume, rank, IP addr
• Public treatment: email + publication in public website
7

Randomization
• Stratification with industry sectors and IP counts
• Pair-wise matching with pre-experimental spam volume
• Re-randomization: 10,000 times and power calculation
8

Treatment channel: email
9

Website: search engine
10
• http://cloud.spamrankings.net

Website: overall stats
11

Website: detail charts
12

System implementation
• Back end: data collector, peer ranker, web generator, MySQL, JSON
• Front end: Google cloud, search engine, analytics
13

Data: CBL and PSBL
14
• A spam blocklist uses spamtraps to collect IP adresses
sending out spams:
• CBL: http://cbl.abuseat.org/
• PSBL: http://psbl.org/
• Spamtrap
• honeypot used to collect spam
• email addresses not for legit communications
• CBL daily avg data
• 8 million IP, 190K netblocks, 21K ASNs, 200 countries

Organizational spam data
15
• IP > netblock > ASN > organization
• IP > netblock: IP lookup
• netblock > ASN: Team Cymru
• ASN > org: algorithm + manual inspection
• Organization data from LexisNexis
• 7919 U.S. organizations identified
• Industry codes: SIC, NAICS
• Public/private, # employees

Org level spam volume and IP address
16

Industry sectors
17

Industry level spam volume/host
18

Hypothesis development
1. Information disclosure effect
2. Publicity effect
3. Pre-experimental security level
4. Industry competition level
19

Info sharing and publicity effects (H1, 2)
20

Large spammers (H3)
21

Competition (H4)
22

Empirical analysis summary
1. Private info sharing doesn’t work
2. Publicity matters
3. Organizations with (1) large spam, (2)
less competition reacted
4. Peer effect exists after the treatments.
Stronger with treatment groups.
23

Robustness check
1. Placebo test: change experiment time
2. Subsample analysis: only include
moderate spammers
3. Alternative pre-experimental spam
measure: 6, 4, 2, months
4. Control variables
24

Directions
1. Robust security evaluation: spam,
phishing, DDoS, etc.
2. Different environment: China, Korea
3. Treatment channel: social media
4. Cybersecurity insurance
5. Cloud security
25

Thank you!
Contact: gene@cs.utexas.edu
26

References (1)
[1] Adelsman, Rony M., and Andrew B. Whinston (1977). "Sophisticated voting with information
for two voting functions." Journal of Economic Theory 15, no. 1: pp. 145-159.
[2] Anderson, Axel, and Lones Smith. "Dynamic Deception." American Economic Review 103, no.
7 (2013): 2811-47.
[3] Anderson, Ross (2001). "Why information security is hard: An economic perspective." IEEE
Computer Security Applications Conference, pp. 358-365.
[4] Aral, Sinan, and Dylan Walker. "Identifying influential and susceptible members of social
networks." Science 337, no. 6092 (2012): pp. 337-341.
[5] Arora, Ashish, Ramayya Krishnan, Anand Nandkumar, Rahul Telang, and Yubao Yang (2004).
"Impact of vulnerability disclosure and patch availability-an empirical analysis." Workshop on
Economics of Information Security, vol. 24, pp. 1268-1287.
[6] Bauer, Johannes, and Michael van Eeten (2009). “Cybersecurity: Stakeholder incentives, externalities,
and policy options.” Telecommunications Policy, Vol. 33, pp. 706-719.
[7] Blei, David M., Andrew Y. Ng, and Michael I. Jordan (2003). "Latent dirichlet allocation."
Journal of Machine Learning Research 3: pp. 993-1022.
[8] Bratko, Andrej, Gordon V. Cormack, Bogdan Filipic, Thomas R. Lynam, and Blaz Zupan
(2006). Journal of Machine Learning Research 6: pp. 2673-2698.
[9] Bruhn, Miriam, and David McKenzie (2008). "In pursuit of balance: Randomization in practice
in development field experiments." World Bank Policy Research Working Paper Series.
[10] Casey, Eoghan (2011). Digital evidence and computer crime: Forensic science, computers and
the Internet. Academic Press.
[11] Cormack, Gordon V., and Thomas R. Lynam (2007). “Online supervised spam filter evaluation.”
ACM Transaction on Information Systems, Vol. 25(3)
27

References (2)
[12] D’Arcy, John, Anat Hovav, and Dennis Galletta (2009). "User awareness of security countermeasures
and its impact on information systems misuse: A deterrence approach." Information
Systems Research 20, no. 1: pp. 79-98.
[13] Denning, Dorothy E. (1987). “An intrusion-detection model.” IEEE Transactions on Software
Engineering, Vol. 13(2): pp. 222-232.
[14] Dharmapurikar, Sarang, Praveen Krishnamurthy, and David E. Taylor (2003). “Longest prefix
matching using bloom filters.” Proceedings of the ACM SIGCOMM Conference: pp. 201-212.
[15] Dice, Lee R. (1945). “Measures of the amount of ecologic association between species.” Ecology
26(3): pp. 297-302.
[16] Duflo, Esther, Rachel Glennerster, and Michael Kremer. "Using randomization in development
economics research: A toolkit." Handbook of development economics 4 (2007): 3895-3962.
[17] Fracassi, Cesare (2014). "Corporate finance policies and social networks." In AFA 2011 Denver
Meetings Paper.
[18] Festinger, Leon. "A theory of social comparison processes." Human relations 7, no. 2 (1954):
117-140.
[19] Gal-Or, Esther, and Anindya Ghose (2005). "The economic incentives for sharing security
information." Information Systems Research 16, no. 2: pp. 186-208.
[20] Graham, Bryan S. (2008). "Identifying social interactions through conditional variance restrictions."
Econometrica 76, no. 3: pp. 643-660.
[21] Harper, Yan Chen, F. Maxwell, Joseph Konstan, and Sherry Xin Li. "Social comparisons and
contributions to online communities: A field experiment on movielens." The American economic
review (2010): 1358-1398.
[22] Harrison, Glenn W., and John A. List (2004). "Field experiments." Journal of Economic Literature:
pp. 1009-1055.
[23] Kugler, Logan (2014). “Online Privacy: Regional Differences.” Communications of the ACM,
Vol. 58 No. 2, pp. 18-20.
28

References (3)
[24] Krebs, Brian (2014). Spam Nation: The Inside Story of Organized Cybercrime - from Global
Epidemic to Your Front Door. Sourcebooks, Inc.
[25] Lee, Wenke, and Salvatore J. Stolfo (1998). “Data mining approaches for intrusion detection.”
Proceedings of 7th USENIX Security Symposium.
[26] Levchenko, Kirill, Andreas Pitsillidis, Neha Chachra, Brandon Enright, Márk Félegyházi, Chris
Grier, Tristan Halvorson, Chris Kanich, Christian Kreibich, He Liu, Damon McCoy, Nicholas
Weaver, Vern Paxson, Geoffrey M. Voelker, and Stefan Savage (2011). "Click Trajectories:
End-to-End Analysis of the Spam Value Chain." IEEE Symposium on Security and Privacy.
[27] Moore, Tyler and Richard Clayton (2011). "The Impact of Public Information on Phishing
Attack and Defense." Communications & Strategies 81.
[28] Morgan, Kari Lock, and Donald B. Rubin (2012). "Rerandomization to improve covariate
balance in experiments." Annals of Statistics 40, no. 2: pp. 1263-1282.
[29] Popadak, Jillian A. (2012). "Dividend Payments as a Response to Peer Influence." Available
at SSRN 2170561, http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2170561.
[30] Pitsillidis, Andreas, Chris Kanich, Geoffrey M Voelker, Kirill Levchenko, Stefan Savage (2012).
“Taster’s choice: A comparative analysis of spam feeds.” Proceedings of the 2012 ACM Internet
Meassure Conference: pp. 427-440.
[31] Rao, Justin M., and David H. Reiley (2012). "The economics of spam." Journal of Economic
Perspectives 26, no. 3: pp. 87-110.
[32] Roesch, Martin (1999). “SNORT: Lightweight intrusion detection for networks.” Proceedings
of 13th Large Installation System Administration Conference, pp. 229-238.
[33] Rothschild, Michael, and Joseph Stiglitz (1992). “Equilibrium in competitive insurance markets:
An essay on the economics of imperfect information.” Springer Netherlands.
[34] Sahami, Mehran, Susan Dumais, David Heckerman, and Eric Horvitz (1998). “A Bayesian
approach to filtering junk e-mail.” Learning for Text Categorization 62: pp. 98-105.
29

References (4)
[35] Shue, Kelly (2013). "Executive networks and firm policies: Evidence from the random assignment
of MBA peers." Review of Financial Studies 26, no. 6: pp. 1401-1442.
[36] Tang, Qian, Leigh Linden, John S. Quarterman, and Andrew B. Whinston (2013). “Improving
Internet security through social information and social comparison: A field quasi-experiment.”
In Workshop on the Economics of Information Security.
[37] Taylor, Robert W., Eric J. Fritsch, and John Liederbach (2014). Digital crime and digital
terrorism. Prentice Hall Press.
[38] Taylor, Shelley E., and Marci Lobel (1989). "Social comparison activity under threat: downward
evaluation and upward contacts." Psychological review 96, no. 4: p. 569.
[39] van Eeten, M., H. Asghari, J. M. Bauer, and S. Tabatabaie (2011). "Internet service providers
and botnet mitigation: A fact-finding study on the Dutch market." Delft University of Technology.
[40] Wood, Dallas, and Brent Rowe (2011). "Assessing home Internet users’ demand for security:
Will they pay ISPs?" Workshop of Economics of Information Security.
30

Designing Cybersecurity Policies with Field Experiments

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (11)

Semelhante a Designing Cybersecurity Policies with Field Experiments

Semelhante a Designing Cybersecurity Policies with Field Experiments (20)

Mais de Gene Moo Lee

Mais de Gene Moo Lee (10)

Último

Último (20)

Designing Cybersecurity Policies with Field Experiments