You and your colleagues are all doing great things with Splunk. But you seldom come together to share ideas, apps and best practices. This session will help you take Splunk to the next level by helping you establish a Splunk Center of Excellence (CoE) at your organization. The purpose of a COE is simple - to provide Splunk users an informal venue in which they can discuss ideas, diagnose challenges, share innovations and network with peers. This session will share the best practices you need to create and maintain a successful CoE practice.
2. 2
About Me
Clint Locker – clocker@splunk.com
Sr. PS Manager for Midatlantic and Southeast Regions
Started with Splunk 4 years ago, delivered Professional
Services for 2 years (may have deployed your
environment, sorry in advance )
Managed over 400+ Splunk successful deployments
Based in Arlington VA, lived in the area for close to 15
years, wife and I have a 8 month old boy William
2
3. 3
Agenda
• Evolution of Splunk Deployments
• Components of a Splunk COE
• Communication Framework
• Training Recommendations
• COE Assessment
4. 4
Splunk Deployments Evolve Over TimeSplunk Deployments Evolve Over Time
4
Expansion
Download
Workgroup
Enterprise
Deployment
• Enterprise standard
• Large number of users
• Many different use cases
• Many different users
• MultipleSplunk
deployments
• More sites
• More geographies
• More data sources
More data volume
InitialUser
• Specific use case
• Specific users
5. 5
Turning Machine Data Into Operational Intelligence
Search
and
Investigate
Search and
Investigate
Proactive
Monitoring
and Alerting
Operational
Visibility
Real-time
Business
Insight
Proactive
Reactive
6. 6
What is COE?
A Center of Excellence refers to a team, a shared
facility or an entity that provides leadership,
evangelization, best practices, research, support
and/or training for a focus area.
7. 7
Splunk Center of Excellence
Goals
Provide Splunk technical oversight
Drive and communicate best practices
Facilitate data on-boarding, user on-
boarding, run book documentation
Provide expertise focused on enablement
Deliver support services to Splunk
consumers
8. 8
Key Success Factors
Program Management Office
Executive sponsorship
Project planning, communication, and
process
Success Criteria Clearly Defined
Business requirements, use cases
Reports, alerts, dashboards
Create Deployment Plan
Architecture
Data/App on-boarding
Resourcing, staffing, training plan
Communications
Regular cadence from technical to executiv
teams
Quarterly Business Reviews
Sustainment
Establish Splunk Center of Excellence
9. 11
COE Sample Benefits
11
Challenge Solution Benefit
What is on your mind? How can the COE help? Measure, improve, let us know!
Lack of general knowledge and
internal best practices for Splunk
led to increased support calls.
30 minutes devoted to addressing
end user needs and education of
internal processes.
Reduction of Splunk Support
interactions by over 25% and faster
time to value.
A flurry of new dashboards led to
decreased performance of system.
Focused training sessions and
advanced techniques education.
Elimination of over 24 redundant
panels and better overall system
performance.
Rapid adoption of Splunk led to a
severe backlog to on-board desired
data targets.
Open discussion on process
improvement, standards and
requirements for new sources.
Increased speed and efficiency of
data onboarding process from 2
weeks to 48 hours on average.
10. 12
Components of a Splunk COE
Architecture &
Infrastructure
Operations
Supporting
Tools
Staffing
Data
On-Boarding
User
On-Boarding
Inform
11. 13
Architecture & Infrastructure
Appropriate hardware sizing for indexing and search load
Physical cores only, hyperthreading does not count
SSD provides significant performance advantage
High performance storage
IOPS are critical
In distributed environments, dedicated IOPS are not cumulative
Measure with Bonnie++, SplunkIT, IOPS App
Current Splunk version
Clear upgrade path and process
Proactive capacity planning
Understand unit of scale for hardware
Map growth curve for data and users
12. Best Practice – Service Levels
Characteristics Staging Class C Class B Class A
Infrastructure Shared Shared Federated Dedicated
Use Case Value Low
(testing)
Low
(discovery)
Medium
(visible, supporting
tools)
High
(revenue/service
impacting)
Retention Short
(2-4 weeks)
Short
(1-3 months)
Medium
(3-6 months)
Long
(6-12 months)
Security/Access Basic Basic Moderate Strong
Chargeback None Simple Mixed Complex
SLA None Lowest Moderate High
Geography Single Single Multiple Single/Multiple
HA/DR None None Partially Resilient Fully Resilient
13. 15
Operations & Supporting Tools
Configuration Management
Common: Puppet/Chef
Splunk: Deployment Server
Change Management
Version control
Service ticketing
Deployment
System Health Monitoring
System capacity and
performance
Splunk tools: Unix App,
Windows App, VMware App,
NetApp App
Splunk Health Monitoring
Splunk on Splunk App
Fire Brigade App
Sanity App
14. 16
Staffing
1
A successful and scalable deployment of
Splunk relies on the orchestration of key
roles and responsibilities, primarily
centered around:
Architecture
Administration
User adoption (Power User)
Application development
15. 17
Splunk Architect Role
1
Responsibility
• Accountable for the design of the Splunk architecture
• Fully understands concepts and best practices for sizing, scaling, and deploying Splunk across your
organization so that performance meets current and future needs
• Works with power users to determine which data sources should be indexed to meet each
department’s needs
Recommendation
• 1 to 2 Splunk Architects
• Part time for < 500GB; 1 Full time for 500GB to 1TB; 2 for >1TB
• Note: if deploying Splunk Cloud, assume only 25% of above resources are required
16. 18
Splunk Admin Role
1
Responsibility
• Maintains the Splunk software and it’s infrastructure for optimal performance
• Adds data sources to the Splunk platform according to Power User needs
• Assist power users with the development of advanced dashboards, alerting and reporting
Recommendation
• 1 to 2 Splunk Admins depending on size of implementation
• Part time for < 500GB; 1 Full time for 500GB to 1TB; 2+ for >1TB
• Note: if deploying Splunk Cloud, assume only 25% of above resources are required
17. 19
Splunk Power User Role
1
Responsibility
• Works with their group to identify opportunities where Splunk can provide value
• Collaborates with the Splunk admin(s) to add new data sources to address their requirements
• Provides basic support for new and existing reports and dashboards to their group from
investigative keyword searches to creating rich reports and visualizations to becoming a Splunk
search ninja!
Recommendation
• 1 part-time power user per user group
18. 20
Splunk Developer Role
2
Responsibility
• Splunk developers are only required if applications are developed on top of the Splunk platform
• Create rich, interactive dashboards and forms, and package Splunk knowledge objects for
distribution across your organization
19. 21
Basic Communication Framework
2
Architect
Admin
Works with power users to determine
which data sources should be indexed
to meet each department’s needs
Scales the Splunk architecture to meet
business demand
Power Users Department Users
Adds data sources to the Splunk
platform according to business needs
Assist power users with the
development of advanced dashboards,
alerting and reporting
Maintains the Splunk SW and it’s
infrastructure for optimal performance
1 Power user per department
Provides basic support for new and existing reports
and dashboards
Works with their group to identify opportunities
where Splunk can provide value
22. 24
Data On-Boarding
Define on-boarding process
for new data sources / apps
Repeatable, documented
process
Provide customer interview
forum or survey
Integrate with service
workflow
New Data Source Request
Provide a data sample
Describe the data’s structure
timestamp | timezone single-/multi-line
sourcetype interesting fields
Describe initial uses for the data
searches | alerts | reports | dashboards
How to collect the data?
UF | syslog | API
How long to retain the data?
Who should have access?
Apply Common information Model
Are there TA’s available?
Validate
23. 25
User On-Boarding
Orientation for new users
Develop training program
Splunk instructor-led online/onsite courses
Get started with Splunk videos
Advancement for experienced users
Continuing education
Splunk workshops
Office Hours
Where to get help?
How to contact <Company> Splunk team
Internal/external email lists, chat group
Splunk Answers
24. 26
Inform
Track Value and ROI
Document Use Cases
Expert Showcases
Internal knowledge sharing
Develop power users
Tip of the Week/Month
Contests
Search competition
Use case drive
Regular Newsletter
Splunk Accelerates Troubleshooting
An expressive troubleshooting dashboard shines a
bright light on any part of the infrastructure exceeding
reasonable performance thresholds.
Less Screwing Up, More Drilling Down
Application and site performance is often dependent on
system performance. Splunk’s monitoring probes
through layers to collect high resolution CPU statistics.
25. 27
Example Meetings
User Group
Splunk in Action
Ask Splunk
Open Office Hours
Splunk Administrators Group
Architecture and Administration topics
Splunk Developers Group
App, UI and API topics
Splunk Lunch & Learn
Education topics
Splunk Support Session
Support case review
Quarterly Business Reviews
Vendor Management Office
26. 28
COE Success – Be Visible & Valuable
• Create a Knowledge Management Portal for Splunk resource
– Publish company specific policies & procedures
– Publish Naming Standards
– Publish Data Onboarding guidelines
– Link to Splunk.com resources
• Aggregate Training Needs from Line of Businesses
• Conduct regular meetings for Line of Business Users
– General User Group for Best Practice Sharing
– Specialized meetings for Administrators, Developers, etc.
– Lunch & Learn Sessions for informal training
2
27. 29
Use Case Documentation (Examples)
Splunk Monitors Proactively for Threat Patterns
Alongside historical trending and analysis for monthly and incident
reports, Splunk alerts the Fraud Detection team to similar patterns
emerging across systems or locales in real time. Email alerts also
promote standardization in capturing and exposing critical
information.
Splunk Secures Access for Independent Forensics
Role-based controls provide shielded views into data. Incident investigations no
longer require highly paid security professionals for pattern tracking and reporting.
Empowering customer service or individual financial institutions to research
independently and securely reduced incident response time from hours to minutes.
Splunk Detects $5M in Attempted Fraud
Correlation by transaction, time and geography identifies all elements in
the infrastructure exposed to nefarious activity originating internally or
externally. In one incident, Splunk’s transaction tracing and geoip
mapping abilities identified 15 banks located in the same region
exhibiting a similar fraud pattern. The activity was tracked to a single
shared data processing vendor which had been compromised.
28. 30
Partner with Splunk Teams
Account Team
Account Manager
Sales Engineer (SE)
Specialists (Security, IT SI, etc.)
Support Team
Designated Support Engineer (DSE)
Customer Success Manager (CSM)
Education
Standard curriculum
(online/onsite)
Boot camps
Customized curriculum
Professional Services Team
Project-based (e.g. Deployment,
Health Checks, Upgrades, App
Development)
Technical Advisory Services (TAS)
Center of Excellence Advisory
Services
Customer Advisory & Success
Teams (CAST)
Dedicated Splunk Advisory
Engineers
Faster time to value and adoption
29. 31
Splunk User Groups
Community driven
Bootstrapped by Splunk
Locally every 2-3 months
SplunkLive!
Worldwide customer events
Technical workshops for beginner and advanced users
Local events held yearly
Annual Worldwide Users Conference
September 26-29, 2016 in Orlando FL, Disney World
3+ days, 130+ sessions, 4000+ enthusiasts
Splunk Answers Desk, SplunkBase Lab, Chalk Talks, Search Party, Hackathon
Get Social with Splunk Events
3
www.splunk.com > Events
30. SEPT 26-29, 2016
WALT DISNEY WORLD, ORLANDO
SWAN AND DOLPHIN RESORTS
• 5000+ IT & Business Professionals
• 3 days of technical content
• 165+ sessions
• 80+ Customer Speakers
• 35+ Apps in Splunk Apps Showcase
• 75+ Technology Partners
• 1:1 networking: Ask The Experts and Security
Experts, Birds of a Feather and Chalk Talks
• NEW hands-on labs!
• Expanded show floor, Dashboards Control
Room & Clinic, and MORE!
The 7th Annual Splunk Worldwide Users’ Conference
PLUS Splunk University
• Three days: Sept 24-26, 2016
• Get Splunk Certified for FREE!
• Get CPE credits for CISSP, CAP, SSCP
• Save thousands on Splunk education!
32. 34
Splunk Architect Training
Splunk
Architect(s)
Using
Splunk
Splunk
Administration
Searching and
Reporting
Creating
Knowledge
Objects
Advanced
Searching &
Reporting
Developing
Apps with
Splunk
Developing
with Splunk
SDKs
• # name
• # name
= Splunk training completed= Required = Optional = Training required but not completed = Optional training not completed
Instructions: List the
names and color
code the cells as
green, red or leave
blank, based on
legend below
33. 35
Splunk Admin Training
Splunk
Administrator(s)
Using
Splunk
Splunk
Administration
Searching and
Reporting
Creating
Knowledge
Objects
Advanced
Searching &
Reporting
Developing
Apps with
Splunk
Developing
with Splunk
SDKs
• #name
• #name
• #name
= Splunk training completed= Required = Optional = Training required but not completed = Optional training not completed
Instructions: List the
names and color
code the cells as
green, red or leave
blank, based on
legend below
34. 36
Splunk Power User Training
Splunk
Power User(s)
Using
Splunk
Splunk
Administration
Searching and
Reporting
Creating
Knowledge
Objects
Advanced
Searching &
Reporting
Developing
Apps with
Splunk
Developing
with Splunk
SDKs
Server Team
• # name
Network Team
• # name
Middleware Team
• # name
DBA Team
• # name
App Support Team
• # name
App Development
• # name
Security Team
• # name
= Splunk training completed= Required = Optional = Training required but not completed = Optional training not completed
Instructions: List the
names and color
code the cells as
green, red or leave
blank, based on
legend below
35. 37
Splunk Developer Training
Splunk
Developer(s)
Using
Splunk
Splunk
Administration
Searching and
Reporting
Creating
Knowledge
Objects
Advanced
Searching &
Reporting
Developing
Apps with
Splunk
Developing
with Splunk
SDKs
• # name
• # name
• # name
• # name
= Splunk training completed= Required = Optional = Training required but not completed = Optional training not completed
Instructions: This slide is optional and only applies IF there are plans to develop applications on top of Splunk.
List the names and color code the cells as green, red or leave blank, based on legend below
36. 38
Your Splunk COE
Splunk
Architect
Doug
Splunk
Administrator
Kevin
Splunk
Developer
Suzie
UX Admins
Power User
Bob
Network
Power User
Mark
DBA
Power User
Dave
ecommerce
Power User
Tony
example
= Fully Trained = Partially Trained = Not assigned
Splunk
Developer
Todd
Instructions: add / remove boxes
as needed. Include existing and
future user groups. Color code
each box based on legend below
37. 39
Splunk COE Recommendations
Roles Assignments
• A
• B
• C
Required Training
• A
• B
• C
architect
admin
developer
power user
Instructions: add recommendations to address role gaps with current and
future user groups. 1 person may carry more than 1 role, however Power
Users are usually different from team to team.
Instructions: add recommendations to address training gaps for current
and future user groups.
These are the milestone stages of Splunk’s growth within most organizations.
Can I get a show of hands on how many people are here in the first stage. Wow, hold on because you have a fun ride ahead of you. I would grab some business cards from the people around you.
…keep going through each stage.
We’re headed to the East Coast!
2 inspired Keynotes – General Session and Security Keynote + Super Sessions with Splunk Leadership in Cloud, IT Ops, Security and Business Analytics!
165+ Breakout sessions addressing all areas and levels of Operational Intelligence – IT, Business Analytics, Mobile, Cloud, IoT, Security…and MORE!
30+ hours of invaluable networking time with industry thought leaders, technologists, and other Splunk Ninjas and Champions waiting to share their business wins with you!
Join the 50%+ of Fortune 100 companies who attended .conf2015 to get hands on with Splunk. You’ll be surrounded by thousands of other like-minded individuals who are ready to share exciting and cutting edge use cases and best practices. You can also deep dive on all things Splunk products together with your favorite Splunkers.
Head back to your company with both practical and inspired new uses for Splunk, ready to unlock the unimaginable power of your data! Arrive in Orlando a Splunk user, leave Orlando a Splunk Ninja!
REGISTRATION OPENS IN MARCH 2016 – STAY TUNED FOR NEWS ON OUR BEST REGISTRATION RATES – COMING SOON!