Analyzing massive datasets in the security domain The ability to analyze massive amounts of data—both structured and unstructured—has quickly become all but mandatory for large organizations that must effectively protect themselves against malicious activity. This exponential growth in available data has created many challenges, particularly in terms of analysis and actionable intelligence in the security domain. It has also created a unique opportunity for developers to look beyond the basic functionality of their application and develop apps that are integrated more seamlessly into the world of big data.

1. Table of contents
Analyzing massive datasets in the security domain......................................................................................................... 2
HP ArcSight ESM and the Common Event Format............................................................................................................. 2
Sample use case: Correlating structured and unstructured data using HP ArcSight ESM and HP IDOL................. 2
Social media monitoring for negative sentiment and insider threat.............................................................................. 3
Social media monitoring for hacktivist threats.................................................................................................................. 5
About HP.................................................................................................................................................................................... 6
Technical white paper
HP ArcSight ESM and HP IDOL
for Security Analytics

2. 2
Analyzing massive datasets in the security domain
The ability to analyze massive amounts of data—both structured and unstructured—has quickly become all but
mandatory for large organizations that must effectively protect themselves against malicious activity. This exponential
growth in available data has created many challenges, particularly in terms of analysis and actionable intelligence in the
security domain. It has also created a unique opportunity for developers to look beyond the basic functionality of their
application and develop apps that are integrated more seamlessly into the world of big data.
Today’s security analysts typically rely on tools such as Security Information and Event Management (SIEM) systems and
log management solutions, both of which focus primarily on the collection and correlation of real-time audit logs from
network devices, operating systems, and applications. By utilizing the tools available in the HAVEn platform, such as the
Common Event Format, developers can greatly enhance the usefulness of their application’s audit data to help enable
security analysts to detect threats.
However, even with increased interoperability, there is a growing need to complement these solutions with more
extensive analytics to identify anomalies and other suspicious activity as attacks become increasingly sophisticated.
Combining an SIEM, like HP ArcSight ESM, with HP IDOL’s unstructured data analytics capabilities together with
applications that have been designed with interoperability in mind, allows organizations to gather the actionable security
intelligence necessary in today’s complex threat landscape.
HP ArcSight ESM and the Common Event Format
HP ArcSight ESM is the premier security event manager that enables organizations to store, analyze, and correlate
millions of events for security event monitoring, from compliance and risk management to security intelligence and
operations. ArcSight ESM sifts through millions of log records, correlates them, and provides identity and asset context
to find the critical events that matter in real time, via dashboards, notifications, and reports, enabling you to accurately
prioritize security risks and compliance violations.
The Common Event Format (CEF) is an open log management standard, created to simplify log management challenges.
It uses a standardized format allowing you to easily collect and aggregate data for analysis by an enterprise log
management system. CEF is an extensible, text-based, high-performance format designed to support multiple device
types and applications in the simplest manner possible. Specifically, CEF defines a syntax for audit log records comprised
of a standard header and a variable extension, formatted as key-value pairs. This format contains the most relevant
event information, making it easy for event consumers to parse and make use of the data.
Sample use case: Correlating structured and unstructured data using HP
ArcSight ESM and HP IDOL
One of the many sources of unstructured data available to an organization comes from social media outlets such as
Twitter, Facebook, Instagram, LinkedIn, GlassDoor, and more. Traditionally, these data streams have been inaccessible
by most SIEM platforms, however much of this information could provide useful insight into pending threats, especially
when combined with structured audit data from network devices, operating systems, and applications.
In this use case, IDOL is used to analyze unstructured social media data to detect negative sentiment towards an
organization. When it finds an email or social media thread, for example, containing negative sentiment, IDOL generates
an event in the Common Event Format and sends it to ArcSight ESM. IDOL can also analyze information to provide
additional business context regarding communications going in or out of an organization. For example, IDOL can analyze
the body of an email, attachments, and recipients to provide additional context beyond the subject line.
In addition, the integration between ArcSight ESM and HP IDOL is bi-directional. An analyst using ArcSight can ask IDOL
for additional information regarding an event they have seen in ArcSight, and IDOL can send CEF events to ESM. This can
be a powerful tool in facilitating an analyst’s investigation into an incident.
Technical white paper | HP ArcSight ESM and HP IDOL for Security Analytics

3. Unstructured data Structured data
HP IDOL HP ArcSight ESM
Alerts ESM to target negative
sentiment communications
and threat intelligence
IDOL provides additional
business context for suspicious
communications
Display to analyst the full
content of communications and
threat intelligence
HTTPS HTTPS
API queryCEF
CEF
CEF
Security devices (FW, IDS, etc.)
Identity and Access Management
Applications
Email, files
Social media, chat sessions
Websites, audio/video
HP ArcSight ESM and HP IDOL
3
Architecture
Social media monitoring for negative sentiment and insider threat
In this scenario, a current employee has posted a review on an employer review website expressing dissatisfaction with
their current employer and the possibility of leaving to go work for a competitor. IDOL has detected this post and sent an
event in CEF to ESM where a correlation rule is triggered and an analyst is alerted to the incident.
IDOL CEF Event:
CEF:0|IDOL|IDOL|1.0|event:0001|IDOL Category Event|0|cfp1=76.42 cfp1Label=Weight cs2=Recipe cs2Label=IDOL
Category cs3=c6c7f0054a7fa385817c13a9605656b6 cs3Label=IDOL MessageID cs4=Negative cs4Label=Sentiment
cs5= cs5Label=Sendmail MessageID cs6= cs6Label=FROM cs7= cs7Label=TO cs8= cs8Label=Handle cs9=
cs9Label=FacebookProfile cs11=http://www.employerreview.com/Reviews/Employee-Review-KobaltSystems-1783.
htm cs11Label=URL
The analyst can access the URL from the event within ArcSight and pull up the post for more information.
Technical white paper | HP ArcSight ESM and HP IDOL for Security Analytics

4. 4
Unfortunately, at this point the analyst does not know who posted this, as these reviews are anonymous. Using HP
ArcSight Identity View and logs collected by ArcSight from the proxy, it can be determined that at approximately the same
times as this post was created, the user jsmith, who is a presales engineer, visited this exact post.
In addition to this, IDOL has sent ArcSight an alert identifying a potentially suspicious email containing source code. Upon
further investigation within ArcSight, using logs collected from Exchange, the analyst determines that this same user
emailed this source code to his personal account. Given his earlier posting to the employer review site, it may be that
this employee is looking to take some code with him, if he potentially leaves and joins a competitor. Using these tools, an
analyst can detect this and avoid a potentially devastating scenario.
Technical white paper | HP ArcSight ESM and HP IDOL for Security Analytics

5. 5
Social media monitoring for hacktivist threats
In this scenario, IDOL is configured to crawl a social media feed for threads containing an
organization’s name and malicious activities. When a thread is detected, a CEF event is sent to
ArcSight ESM alerting an analyst of this potential DDOS attack along with the poster’s handle and the
sentiment evaluation of the post.
Sample IDOL Configuration File:
[SocialMedia_Search0]
Query=Kobalt Systems
LuaConfigureScript=C:AutonomyIDOL10SocialMediaConnectorlua_scriptsconfigure_
socialmedia_search.lua
ConnectionLibrary=connectionSocialMedia.dll
type=popular
USER=someUser
CONSUMER_KEY=xbp0rUbSxM8kSWuSKV1234
CONSUMER_SECRET=OGGnKPVuLinR00GIvEcyrBIvwSzg69Dg3qPb5c1234
ACCESS_TOKEN=891363228-pyKqh02EfuqNgfyu16iUlLDqkr1EbHthjky65vWv
ACCESS_TOKEN_SECRET=0F4AXpYJxl3Yhqt8yIApCMiHtOZlVsyGry3YRZf3b4
IndexDatabase=SocialMedia
IDOL CEF Event:
CEF:0|IDOL|IDOL|1.0|event:0001|IDOL Category Event|0|cfp1=90.18 cfp1Label=Weight cs2= DDOS
cs2Label=IDOL Category cs3=1e2c814ab194af80e0743fac572ba26e cs3Label=IDOL MessageID
cs4=Negative cs4Label=Sentiment cs5= cs5Label=Sendmail MessageID cs6= cs6Label=FROM
cs7= cs7Label=TO cs8=hakdplnt cs8Label=SocialMediaHandle cs9= cs9Label=FacebookProfile
cs10= cs10Label=User Internal Email
The analyst can access the URL from the ArcSight console and see the thread.
Technical white paper | HP ArcSight ESM and HP IDOL for Security Analytics

6. © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only
warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein
should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.
20131120_PI_WP_HP_ArcSight_ESM_IDOL_for_Security_Analytics
Share with colleagues
Sign up for updates
hp.com/go/getupdated
Technical white paper | HP ArcSight ESM and HP IDOL for Security Analytics
Additionally the poster has identified a vulnerable system and posts this information as well.
By combining the ability to analyze unstructured data using HP IDOL, the powerful correlation engine
of ArcSight ESM, and the enhanced interoperability made possible by the Common Event Format, the
analyst now has actionable information to use to potentially block this attack. This level of analysis
has traditionally not been available to organizations where they relied primarily on SIEM solutions
alone, but it is now possible using the HP HAVEn platform.
About HP
HP creates new possibilities for technology to have a meaningful impact on people, businesses,
governments and society. The world’s largest technology company, HP brings together a portfolio
that spans printing, personal computing, software, services and IT infrastructure to solve customer
problems.
More information about HP (NYSE: HPQ) is available at hp.com.