We explore "Metrics, mstats and Me: Splunking Human Data” and also have some insights into the KV Store and javascript use in dashboards. We’ll also re-cover the conf18 updates for those who couldn’t attend our last session.

1. © 2018 SPLUNK INC.© 2018 SPLUNK INC.
Splunk User Group Edinburgh

2. © 2018 SPLUNK INC.
Harry McLaren
● Managing Consultant at ECS Security
● Splunk Enablement Lead & Member of Splunk Trust
● Leader of the Splunk User Group Edinburgh

3. © 2018 SPLUNK INC.
Introduction to ECS Security
Splunk Partner - UK
– Security Consultancy & Managed SOC Provider
– Splunk Revolution Award & Splunk Partner of the Year

4. © 2018 SPLUNK INC.
Agenda
• Housekeeping: Event Overview & House Rules
• Metrics, mstats and Me (Andrew McManus)
• KV Store & Javascript (Mark Hunter)
• Splunk .conf18 Updates (Harry McLaren)
• Security
• IT Ops
• Others (Docker)

5. © 2018 SPLUNK INC.
Splunk [Official] User Group
“The overall goal is to create an authentic, ongoing
user group experience for our users, where
they contribute and get involved”
● Technical Discussions
● Sharing Environment
● Build Trust
● No Sales!

6. HUMAN DATA TO SPLUNK
METRICS, | MSTATS AND ME
EDINBURGH SPLUNK USER GROUP – 22/11/2018
ANDREW MCMANUS – ASSOCIATE SECURITY CONSULTANT - ECS

7. ABOUT MYSELF
• Associate Security Consultant at ECS
• Prior - Senior/Security Operations Center Analyst at ECS
• (Non-Pearson) Credentials: Admin, Sales Rep I
• Know a lot about searches.
• Like to mess about with shiny new Splunk Additions
• Type 1 Diabetic – Since 2001.
• Part Cyborg

8. DIABETES 101? ISN’T THIS A SPLUNK TALK?
• Body uses Insulin to regulate glucose between blood stream and cells
• Type 1 – Something causes destruction of insulin cells in Pancreas, causing deficiency.
• No-one’s sure about exact cause – widely believed to be Auto-Immune related.
• Manual injections required. Manual glucose testing required.
• Type 2 – Resistance to Insulin, normally through diet or environmental aspects.
• Can go into remission with treatment/diet.
• Can use injections or pills to regulate glucose content.

9. DISCLAIMER
• I’m not intentionally advocating treatments or products.
• There are pros and cons to the products/treatments mentioned
• Price, usability, comfort, reaction times…
• Yes, I can eat sugar. Common misconception.
• I shouldn’t, but that’s on me. I crave dessert too much.
• Go to your GP if you have health concerns.

10. MEASUREMENT STEPS (UNTIL 23/07/2018)
• Glucose sample from blood, via a finger pricker.
• Glucose meter takes static snapshot of blood glucose concentration
• Sample taken before major meals, and ad-hoc if required
• Insulin taken as a response to glucose result, or recommended dosage
• Aiming for between 4mmol/l and 10mmol/l glucose concentration.

11. MEASUREMENT STEPS (SINCE 23/07/2018)
• Prescribed Abbott Libre FreeStyle sensor (other sensors available)
• Checks glucose content in interstitial fluid below skin, not blood
• Takes reading every 1m and calculates trending behaviour.
• Retains a rolling 8 hours worth of data on sensor
• Transfers readings to monitoring device, or phone, via NFC

13. A QUICK DEMONSTRATION
• Hope the Pizza and Beer don’t shame my glucose levels…

14. LESS MEDICAL, MORE METRICAL PLEASE!
• Sensor is continuously taking metric data of glucose concentration in body.
• Phone or meter can send this metrics data to a cloud service for doctors to see
• Cloud service provides a export of metric data to local machine.
• Metric data is Machine Data
• Splunk likes Machine Data – Splunk has special metrics gizmos baked in.

15. SPLUNK METRICS
• Meant for “collecting, investigating, monitoring, and sharing metrics from your technology
infrastructure, security systems, and business applications in real time”
• Fast statistical results and visualizations using command Splunk commands
• Can’t search for events, in traditional sense (i.e. security logs).
• Claims: 20x faster than equivalent accelerated log (tstats) and 200x faster than non-
accelerated logs/event data searches.
• What makes up a Metric?

16. SPLUNK METRICS
• Timestamp
• Timestamp of metric
• Metric Name
• dotted namespace i.e. server.www1.response.5xx
• Value
• Numerical data point
• Dimensions
• Metadata to describe data – i.e. AWS AZ, server name, technology name
• Can have multiple dimensions

17. GETTING METRICS IN
• Various methods.
• | mcollect, HEC, statsD, collectD, csv to metric, Insights for Infra App

18. COLLECTD
• https://collectd.org
• Periodically collects system and application performance metrics.
• Point collectd’s write_http module to HEC with collectd_http sourcetype
• Quick Demo – Computer Metrics

19. METRICS FROM LAPTOP
Hint: Don’t be like me. Use Splunk App for Infrastructure
(https://splunkbase.splunk.com/app/3975/)
Sets Collectd up for you.

20. DIABETIC DATA TO METRIC_CSV
• Data needed to be transformed to match metric_csv sourcetype
• Quick dirty Python Script to import csv, transform timestamps and collapse
data to expected fields
• Write to new file and ingest this on a monitor input
• Danger – No “| delete” method for metrics – once it’s in, it’s in.
• Keep in mind if monitoring a file, or one-shotting data in.

21. QUICK CODE REVIEW

22. MCATALOG
• List metric names, hosts and dimensions
• Useful to see what metrics you have in Splunk
• | mcatalog values(_dims) values(host) by metric_name

23. MSTATS
• Run statistical commands on metric values.
| mstats avg(_value) as ”avg_glucose" WHERE metric_name="personal.glucose.historic" AND "index"="diabetic_data"
span=1h
| append [| mstats sum(_value) as total_quick_insulin WHERE metric_name="personal.insulin.rapid.dose.units" AND
index=diabetic_data span=1h]
| append [| mstats sum(_value) as total_carbs WHERE metric_name="personal.carbohydrate.grams" AND
index=diabetic_data span=1h]

24. VIEWING METRICS – METRICS EXPLORER
• App on Splunkbase – will be added to core Splunk eventually.

25. NEW! FROM CONF 2018: METRICS WORKSPACE
• Download from SplunkBase:
• One stop shop for metric discovery, dash-boarding and alerting.
• No SPL required

26. FURTHER READING
• .conf2018:
• Getting logs and metrics into metricstore (https://conf.splunk.com/files/2018/recordings/getting-logs-and-
metrics-fn1888.mp4)
• New Splunk Metrics Workspace Experience (https://conf.splunk.com/files/2018/recordings/exciting-to-be-
announced-fn1508.mp4)
• .conf2017:
• Getting Metrics In: Splunking Metrics – The Right Way (https://conf.splunk.com/files/2017/slides/getting-
metrics-data-in.pdf)

27. ANY QUESTIONS?

28. THE KVSTORE FOR
FUN AND PROFIT*
* Profit not guaranteed **
** Fun not guaranteed either

29. ELEGANT CAT, SITTING
• This is my cat.
• His name is Roran.
• He also answers to “Catface.”
• I call him this because his face bears a quite uncanny
resemblance to the bewhiskered visage of a cat.
• Also ”Roy Cattersley”, “Catweazel” and “The Floofmeister.”
• When my waffle becomes intolerable, think back to his
fluffy coat, his furry paws, his gentle smile. It’ll all be over in
no time.
• One way or another.

30. PART 1: CSV VS KV STORE. FIGHT!

31. EXTENDED CHAT SUMMARISED
• As the title hints, I’m going to talk about the KV Store :
• How KV Store collections differ from CSV collections
• How to quickly deploy KV store collections
• How to take advantages of what they offer
• Quick look at an in-development SimpleXML-extended KV Store dashboard

32. EXPLANATORY CSV SLIDE
• CSV lookup queries look like this:
• index=foo sourcetype=alignment | lookup detectEvilLookup isEvil | where
isEvil=1
• |inputlookup detectEvilLookup where characterClass=“Fighter”
• |inputlookup detectEvilLookup where characterClass=“Fighter” | outputlookup
theworstLookup

33. EXPLANTORY C..ER..KV-STORE SLIDE
• On the other hand, KV Store lookups look like this:
• index=foo sourcetype=alignment | lookup detectEvilLookup isEvil | where
isEvil=1
• |inputlookup detectEvilLookup where characterClass=“Fighter”
• |inputlookup detectEvilLookup where characterClass=“Fighter” | outputlookup
theworstLookup append=f

34. EXPLANATION COMING, STAT
• No real difference in addressing them.
• CSV files reside on indexers, KV Store on search heads.
• CSV files can only append to or replace file ; KV Store can add, upsert,
and delete specific field entries.
• KV Store has REST endpoint access.
• KV Store can enforce data types.

36. ENOUGH! COMPARE SYSTEMS
Collections
Records
Fields
_key
Tables
Rows
Columns
Primary Key

37. EXAMPLE CASES & SITUATIONS
• Better performance with a larger or frequently updated record set
• Any record management system – inventory, control lists, etc
• Preserving application state
• Scratchdisk
• (Field acceleration!)
• Porting

38. PART 2: SET-UP
AND IMPLEMENTATION

39. ELUCIDATE CLEAR STEPS
• System set-up tasks
• Configuring a collection
• Dashboards and logic

40. EGADS! CAT SHENANIGANS.
• I thought you might like to be reminded of Catface.

41. http://downloads.jordan2000.com/splunk

42. EASILY CONFIGURED – SEE!
• What do you need?
• Two conf files: collections.conf and transforms.conf in a search head app
• You can do this in the GUI, but we are not teh n00blets
• el oh el

43. EXAMPLE CONF SETUP (1)

44. EXAMPLE CONF SETUP (2)

45. EGREGIOUSLY CATASTROPHIC
SUGGESTION
• Let’s live dangerously and try jumping straight to an example.

46. PART 3: FROM THEORY TO
(BADLY IN NEED OF) PRACTICE

47. EXPECTED CONVERSATIONAL SLOG
• The problem
• The config
• The dashboard
• The javascript

48. EXPLANATORY CONTEXTUALISING
STATEMENT
• Replace an existing inventory and control management system
• Based on copied and pasted excel sheets
• Frequently updated daily
• Potentially large updates
• Referenced by many apps for gatekeeping

49. EXTENDED COMMENTARY SECTION 1
• Collection LIVE SHOWING. YOLO!

50. EXTENDED COMMENTARY SECTION 2
• Dashboard LIVE SHOWING! LIVE DANGEROUSLY!

51. EXTENDED COMMENTARY SECTION 3
• Javascript LIVE SHOWING! I’ve run the JOKE into the GROUND!
• Gains over CSV – any?

52. EVENTUALLY (COMING SOON)
• Custom renderer
• Monitoring and troubleshooting tools
• Current client view of MC: ”supping from the very bladder of Satan.”
• A direct quote.*
* Not a direct quote

53. EMBARKING CAREFULLY=SUCCESS
• Some things to be aware of:
• Export and import is everything or nothing
• Use CSV to export and import individual collection. IRONY.
• Auto lookups
• Switch replicate to true in collections.conf stanza
• You’re on the indexers now though
• Filtering with where
• Declare _key

54. THANKS, ALL. THALL.
• Feedback pls

55. © 2018 SPLUNK INC.
Splunk .conf18
Updates
Harry McLaren

56. © 2018 SPLUNK INC.
Introducing Splunk Enterprise Security 5.2
Generally Available: 16/10/18

57. © 2018 SPLUNK INC.
Event Sequencing
Define Attacker Techniques via Multiple Matching Events
▶ The Event Sequencing Engine runs as a real-
time search and listens for incoming notable
events and risk modifiers that are triggered by
correlation searches.
▶ Transitions can also be configured to
aggregate notable events or risk modifiers that
may happen after a transition match is found.

58. © 2018 SPLUNK INC.
Event Sequencing
Define Attacker Techniques via Multiple Matching Events

59. © 2018 SPLUNK INC.
Use Case Library
ES Content Updates Type Function Integrated

60. © 2018 SPLUNK INC.
Investigation Workbench
Two New Artifact Types - File Name & URL

61. © 2018 SPLUNK INC.
Introducing Splunk Phantom Version 4.0
Security Orchestration, Automation, & Response (SOAR) Platform
▶ Clustering support for added performance and redundancy
• Enables Phantom to scale horizontally using additional instances for added performance
and redundancy
▶ Indicator View for threat intelligence style analysis
• Provides a new and important way to visualize security data on the Phantom platform.
Data is presented in the view organized by indicator, versus event, for easier threat-
intelligence style analysis.
▶ Native Splunk search support
• Splunk is now the default search engine shipped with the Phantom product. Users are
able to use their existing or new external Splunk instances to achieve a single source for
security data storage.
Elasticsearch engine remains an external option for those that prefer it.

62. © 2018 SPLUNK INC.
Introducing Splunk User Behaviour Analytics 4.2
Generally Available: 16/10/18
▶ User Feedback for machine learning
models provides anomaly customization and
improved threat detection accuracy
▶ Improved data ingestion performance by
up to 10x, with the new Splunk-to-Kafka UBA
ingestion connector.
Kafka ingestion does not require UBA to run
real-time indexed search queries on core
Splunk, rather uses micro-batched queries.
▶ Native single-sign-on authentication
support for multiple identity providers Okta,
Microsoft ADFS and Ping Identity

63. © 2018 SPLUNK INC.
Introducing Splunk ITSI 4.0
Predictive Analytics for Real-Time Insights
▶ KPI Predictions
We’re excited to deliver deeper insights into a
potential health degradation with KPI
Predictions. These utilize the breadth of data in
the platform to help predict KPIs like customer
experience, application workload, and
infrastructure health, in order to identify issues
or outages in advance.
▶ Predictive Cause Analysis
This new feature helps you drill down into the
specific services underlying a predicted issue to
proactively remediate and resolve it before
customer experience is impacted.

64. © 2018 SPLUNK INC.
Introducing Splunk SmartStore
Cut the Cord by Decoupling Compute and Storage
▶ Allowing compute and
storage tiers to be
independently scaled.
▶ Automatically evaluates
users’ data access
patterns to determine
which data needs to be
accessible for real-time
analytics and which data
should reside in lower
cost, long-term storage.

65. © 2018 SPLUNK INC.
Introducing Dynamic Data: Active Archive
Data Retention Options in Splunk Cloud
▶ Data Management
• Splunk provides complete lifecycle management of the
archive on your behalf and remains the custodian of your
data. Just like your Active Searchable data, Splunk
manages all aspects of archive availability, durability,
security and privacy requirements on your behalf.
▶ Data Restore
• Enables you to request a slice of your data to be restored
back into your Splunk Cloud instance. The entire workflow
is fully integrated into Splunk Web so your archived data is
available at your fingertips with predictable time between
retrieval to search.

66. © 2018 SPLUNK INC.
Other Features!
Selection of Interesting New Releases!
▶ Dark Mode heightens visual contrast within Splunk dashboards.
▶ Workload Management enables users to prioritize the allocation of compute and
memory resources used by Splunk on searches and alerts to ensure users’ most
critical analytics are completed first.
▶ Guided Data Onboarding is a new graphical user interface helping customers
move data into Splunk Cloud or Splunk Enterprise and guiding them through the
best onboarding methodology based on their specific architecture.
▶ Logs to Metrics helps configure and convert log events to metrics, enabling
users to take advantage of breakthrough performance when monitoring and
alerting on metrics with the Splunk platform.
▶ Health Report gives Splunk administrators immediate visibility into the overall
health status of their Splunk environments.

67. © 2018 SPLUNK INC.
Introducing Splunk Next
Splunk Works the Way Your Data Works
▶ Feedback from Splunk Customers
• Make it easier to access data with Splunk no matter where it lives or what format it is in.
• Make it easier to automate the actions and outcomes in order to drive the business forward.
• Make it possible for all kinds of people to ask questions of Splunk and get to answers, no matter
their role or where they might be in the world.
▶ What Does Splunk Next Do For You?
• Ask Questions: Open customers to a broader set of data sources.
• Get Answers: Empower a broader set of customers from IT and Security to Lines of Business.
• Take Action: Operate on data wherever it lives.

68. © 2018 SPLUNK INC.
Splunk Next
Experimental, Pre-release Features (Alpha/Beta)
▶ Splunk Developer Cloud: Write Splunk applications natively in the cloud.
▶ Splunk Business Flow: Analytics-driven approach into customer/user’s interactions and identify
ways to optimize those interactions and processes.
▶ Splunk Data Fabric Search: Seamlessly search across massive amounts of data and federated
searches across multiple instances.
▶ Splunk Data Stream Processor: Refine, modify and adjust data mid-stream and within
milliseconds before the data reaches its destination.
▶ Splunk Cloud Gateway: Secure cloud service with end-to-end encryption for easy mobile
engagement through a simple to install Splunk app for Mobile.
▶ Splunk Mobile: Actionable alerts and mobile-friendly dashboards on mobile devices through our
Splunk Mobile App.
▶ Splunk Natural Language: Query a system and ask question of Splunk without knowing SPL
▶ Splunk TV: View Splunk on any peripheral device instead of having to purchase a dedicated PC
▶ Splunk Augmented Reality: Enjoy direct access to the Splunk dashboard and live augmented
reality Splunk-powered gauges on top of real-world objects.

69. © 2018 SPLUNK INC.
Splunk on Docker
Containers are now a First-Class Citizen
▶ Splunk Support now covers Splunk Enterprise 7.2 deployments in Docker
containers, enabling customers to quickly deploy and scale Splunk based on their
organizations’ demands.

70. © 2018 SPLUNK INC.
Get Involved!
● Splunk User Group Edinburgh
– https://usergroups.splunk.com/group/splunk-user-group-edinburgh.html
– https://www.linkedin.com/groups/12013212
● Splunk’s Slack Group
– Register via http://splunk-usergroups.signup.team/
– Channel: #edinburgh
● Present & Share at the User Group?
Connect:
‣ Harry McLaren | harry.mclaren@ecssecurity.co.uk | @cyberharibu | harrymclaren.co.uk
‣ ECS | enquiries@ecs.co.uk | @ECS_IT | ecs.co.uk