This document discusses log analysis automation challenges, existing log management tools, and proposes a structured log analysis framework. It notes that while existing tools address some automation needs, they ignore the importance of log structure. The proposed framework uses a declarative language to express any log file format, provides rule-based automation scripts, and a flexible data management scheme. Areas of future work include adding more log management capabilities, real-time analysis, optimizing data handling, and improved user interfaces.
2. Functional
Troubleshooting
Conformance
Log Analysis
in Use
Monitoring
Statistical Insight
System Health
3. Log Analysis
Domains
Web server logs
Network logs
Security logs
System logs
Application logs
4. Even with
Manual analysis
expertise,
needs
manual log
acquaintance
analysis is
with format
laborious
Manually Manual analysis
dealing with hinders reusing
vast amount of recurring
log information analysis
is difficult Automation patterns
will save lot
of costs
5. Log Analysis Automation
Challenges
Lack of a standard
• “Universal Format for Logger Messages” - Expired without a successor
• “Syslog” – Serves only a limited range of system logs
Log file corruptions
• Erasing parts of a log file, mixing up multiple log entries, presence of log
entries in wrong order and garbage in the middle of log files
Inappropriate log content
• Problem stems from incorrect judgments of developers regarding the
importance of log entries
Varying log semantics
• Format and the content logged can continue to evolve
Huge sizes of log files
• Log files can easily grow into gigabyte sizes in a commercial environment
10. Conclusions
Existing tools solve a subset of automated log analysis requirements,
but ignore the importance of structure
New declarative language is capable of expressing any log file format
and is resilient to corruptions
The scripting language provides solid infrastructure for rule based
automation
Data management scheme offers flexibility
Current UI generation method is not appropriate
11. Future Work
Add more log management capabilities
Real time analysis
Built-in format declarations for common log formats
Optimize data management module to handle heterogeneous data
efficiently
UI generation based on HTML5
Notas do Editor
Your introductory or title slide should convey the overall “feeling” and focus of your presentation. For instance, I typically present about small-business trends, new business ideas, growth opportunities or other positive trends. In this sample presentation, I’m talking about new business ideas, so I used a sun graphic in this slide template to convey a positive feeling. Personalize this slide template with your company’s logo. To add a logo to all slides, place it on the Slide Master. To access the Slide Master, on the Themes tab of the Ribbon, click Edit Master and then click Slide Master.Disclaimer: You understand that Microsoft does not endorse or control the content provided in the following presentation. Microsoft provides this content to you for informational purposes only; it is not intended to be relied upon as business or financial advice. Microsoft does not guarantee or otherwise warrant the accuracy or validity of this information and encourages you to consult with a business or financial professional as appropriate.RIEVA LESONSKY Founder and President, GrowBiz Media RievaLesonsky is founder and president of GrowBiz Media, a content and consulting company specializing in covering small businesses and entrepreneurship. A nationally known speaker and authority on entrepreneurship, Lesonsky has been covering America’s entrepreneurs for nearly 30 years. Before co-founding GrowBiz Media, Lesonsky was Editorial Director of Entrepreneur Magazine.
I like to speak spontaneously, so I use PowerPoint as an outline to keep me on track. It’s best to keep your PowerPoint text brief, simply reinforcing key points you will talk about at more length. You can use this slide template to convey a series of steps or related points in a short format.
I like to use an off-balance layout to keep things from getting too symmetrical. Customize this slide template graphics of your choice, including photos, clip art, your logo or illustrations. Good photos really help cement an idea in the audience’s mind. This slide is animated to display an appropriate image as you introduce each business type.
A plain old bulleted list can get boring, so use graphics to liven it up. An image that conveys what you’re saying in visual format (like this diagram) can reinforce your ideas in the audience’s mind.
I like to use an off-balance layout to keep things from getting too symmetrical. Customize this slide template graphics of your choice, including photos, clip art, your logo or illustrations. Good photos really help cement an idea in the audience’s mind. This slide is animated to display an appropriate image as you introduce each business type.
Splunk – This is one of the most popular commercial log analysis tools [11]. It comes as a native application for each of the popular platforms. It provides strong search capabilities within log files. Log files from many different sources can be integrated into an analysis. Splunk is capable of identifying common constructs appearing in logs such as timestamps. In addition to indexing logs based on automatically detected log entries it provides functionality for users to create custom indexes too. Indexed log files can be saved as templates so that the index can be used for a similar log file later. Analysis results are displayed in a dashboard with many feature-rich user interface controls. Although Splunk can handle any kind of text log file, it is appropriate for analyzing line logs. It comes with a free version (without expiration) with an upper limit to the total size of log files analyzed in a day.LogRhythm – This is another widely used commercial tool for log analysis [12]. Its' important features are the ability to analyze a huge number of logs at once, automatic detection of interesting log entries, risk-based prioritization of log events, customizable rules, alerts, real-time log monitoring, normalization between different time zones, configurable charting, ability to save investigation data and file integrity monitoring. It has built in capabilities to evaluate log compliance with a number of standards. In addition it has strong intrusion detection capabilities too. In summary, LogRhythm is a sophisticated, enterprise solution.ArcSight Logger – This is a tool for event log collection and reporting [13]. Being a commercial tool ArcSight Logger has the capability to handle event log messages from many different client platforms. The messages can be sent in a varietyof protocols. The tool can handle terabytes of log data efficiently. It classifies log events so that different syntax used across platforms for same kind of log data is made transparent to the user. Searching is possible using plain text, regular expressions or indexed text. ArcSight Logger provides strong reporting capabilities too. Reports can be exported to various formats before saving. Alerts can be defined based on reports. The tool comes with a free evaluation version.loggly – This provides a cloud based log management system [14]. Log files from various sources can be collected to a central place in cloud for analysis. Log entries can be searched and be viewed in a dashboard. Historic data can also be viewed. The tool supports alerting. A free trial version is provided.loglogic – This is another log management infrastructure tool with the capability to collect logs from either enterprise or cloud and provide analysis [15]. Main features include ability to handle data in ranges of petabytes, advanced searching capabilities, dynamic dashboard, detailed reports, alerts, forensics engine, log retention management and compliance reporting.AWStats – This is a free tool that can analyze logs generated by web servers like Apache web server, Internet Information Server, WebStar and some other proxy, wap, ftp, streaming and mail servers [16]. It is a command line tool that uses Perl scripts. It provides usage statistics, user origin information, popularity of pages, HTTP errors, number of favorites on the site, worm attacks detection, etc.SecureVue – This is a situational awareness platform that utilizes logs from various types of assets in an organization such as hosts, network and security devices, applications and databases for capturing important security information [17]. It provides compliance with many security standards.
A plain old bulleted list can get boring, so use graphics to liven it up. An image that conveys what you’re saying in visual format (like this diagram) can reinforce your ideas in the audience’s mind.
I like to use an off-balance layout to keep things from getting too symmetrical. Customize this slide template graphics of your choice, including photos, clip art, your logo or illustrations. Good photos really help cement an idea in the audience’s mind. This slide is animated to display an appropriate image as you introduce each business type.
I like to use an off-balance layout to keep things from getting too symmetrical. Customize this slide template graphics of your choice, including photos, clip art, your logo or illustrations. Good photos really help cement an idea in the audience’s mind. This slide is animated to display an appropriate image as you introduce each business type.
I like to use an off-balance layout to keep things from getting too symmetrical. Customize this slide template graphics of your choice, including photos, clip art, your logo or illustrations. Good photos really help cement an idea in the audience’s mind. This slide is animated to display an appropriate image as you introduce each business type.
I like to use an off-balance layout to keep things from getting too symmetrical. Customize this slide template graphics of your choice, including photos, clip art, your logo or illustrations. Good photos really help cement an idea in the audience’s mind. This slide is animated to display an appropriate image as you introduce each business type.