MuleSoft Manchester Meetup #2 slides 29th October 2019

Manchester, UK | 29 October, 2019
Manchester MuleSoft Meetup #2 -
Monitoring & Logging
Francis
Integration Analyst
Saint-Gobain Building
Distribution
Dave
Lead Solution Engineer
MuleSoft
Kevin
CTO
Ricston

Dave Felcey - Solution Engineer
Anypoint Monitoring
For Resilient Operations

Incident #22664
Failure rate above threshold
Incident #25324
Incident #64243

Is this a
real issue?
Do I even know
how to fix this?

All contents © MuleSoft Inc. 9
Account
Web
Transaction
Order
Account for
SAP
Order
Fulfillment
Notification
for twitter
Order for
.net
Product
for NetSuite
Product
for SAP
Notification
for gmail
Notification
Notification for
facebook
Product
for MySQL
Product
Notification
for twilio
SAP
SAP
Account for
MySQL
!
Product
for SAP
!
Account for
SFDC
Meanwhile…

All contents © MuleSoft Inc.
What was a simple…
12
Technical
issue
Customer
issue
Business
issue

How can this happen?
14
Alert storms Distrust in the data
Uncertainty on
how to resolve
Lack of APM and logging strategy in place

Which means…
15
Team morale is low
Endless alerts
and fire drills
MTTR is too long
Issues aren’t
resolved quickly
TCO is high
High cost from
licensing, staffing a
team, and long MTTR

So how do you...
16
Get alerts about
real issues
Store and access
relevant data
Know what to
resolve and how
Put an APM and logging strategy in place

5 steps to
operationalize monitoring

1. Establish what you want
to track and how

A good place to start is by measuring…
19
● Inbound and outbound traffic
statistics (eg. tx volume)
● Failed requests or transactions
● Request or transaction response
time
● External dependent projects
● Specific sub-components of your
app (eg. APIs, transactions,
flows, etc.
● Proxies for adoption and
business impact (eg. volume of
billing transactions, etc.)
● Compare performance/errors
between environments (e.g.
pre-prod and prod)
API Manager Feature

Then work up to…
time
flows, etc.
● Performance/errors across
environments (eg. pre-prod vs.
prod)
Titanium Feature

As a bonus
time
flows, etc.
● Performance/errors across
environments (eg. pre-prod vs.
prod)
Rely on one source of truth for all your environments --
cloud, onprem or hybrid

2. Build a targeted view and a
process around reviewing it

When building your dashboard
BONUS!
Don’t start from scratch

Increasing adoption of custom dashboards
● Save and share them
● Walk the team through the
dashboards
● Build understanding and
alignment
● Link them from relevant places
● Establish a naming convention
so that it’s easy to find
● Make it easy to find
● Build a process around it
Titanium Feature

How do you build a process?
26
• Set regular review
cadence
– Ship to production
– End of every sprint
• Share relevant metrics on
a set cadence with
management chain
• Make it the first place to
go to during an incident

All contents © MuleSoft Inc. 28

Red alerts
Active incident
Yellow alerts
Prevent incidents
Types of alerts to set up
29

Alerting best practices
● Review historical data to determine the right threshold
● Set the right level of frequency the data is reviewed
● Use rolling averages or percentiles to smooth out noisy spikes in
your data
● Operationalize the application of alerts as new APIs are built
When setting up alerts
30

Standardise alert message formats
31
Make Them
● Informative - use embedded
properties and links
● Instructive - give guidance,
but automate if possible
● Simple and easy to read -
don’t include stack traces

Resource Alerting
● Resource Threshold exceeded
● Worker not responding or
deployment failed
● Can use rolling averages or
percentiles to smooth out
noisy spikes in your data
● Notify anyone by email
Resource Alerting
32Base Feature
Active

Functional Alerts
● Check end-to-end flow
● Operate as beacon
● Wide range of notification
methods, like Slack, Email,
PagerDuty, NewRelic etc.
Functional Alerting
33
Active
Base Feature

API Alerting
● API SLA Threshold
● Policy violations
● Can use rolling averages or
● Notify Anypoint user via
email
API Alerting
34API Manager Feature
Preventative

Custom Alerts
● Alert on wide range of
metrics
● Set triggers using wide range
of filters
● Use rolling averages or
● Create email notifications
Create custom alerts for your use cases
35Titanium Feature
Preventative

Service Reports
● Analyse services over time
and identify where
weaknesses lie
● Summarise failures,
performance etc
● Exportable as CSV
● Multiple timeframes
Analyse system performance over time
36Titanium Feature
Preventative

Define business-level metrics
38
Revenue Order volume
Customer
satisfaction

● Identify business-level
metrics and goals
● Track impact of APIs on
business performance in
real time
● Build business-facing
dashboards which you can
report out
● Leverage insights to
optimize your business
and API programs
Track and optimize impact on business
39Titanium Feature

5. Implement a logging
strategy

Defensive logging is not a strategy
41

Log identifiable data
● Order ID
● User ID
● Correlation IDs for tracing across
distributed APIs
Log diagnostic data
● Success & failure states
● REsponse codes
● Errors and exceptions
● System state
Logging best practices
42

Log diagnostics
● Aggregated log searches
● Trace Order ID across multiple logs
● Identify cross services issues faster
Aggregated log search
43Titanium Feature

Log Points
● Dynamically capture HTTP traffic
● Select key attributes
● Enable/disable in production with
minimal impact to performance
●
Dynamically debug production Web traffic
44Titanium Feature

Storing your logs
45
Multi-tenanted Single-tenanted
1 GB Petabyte
30 days Forever

Securing your logs
46
New values for the same data Same values for the same data
Encryption Tokenization
Not format-preserving Format-preserving

Key takeaways
48
1. Establish what you want to track and how
2. Build a targeted view and a process around reviewing it
3. Set up alerts, including red and yellow
4. Implement a logging strategy
5. Measure business impact
Remember: Perfection isn’t key! Getting started is.
Start this week with 1 & 2

Deep dive into logging
Francis Edwards, Integration Analyst at Saint-Gobain Building Distribution

Logging – Deep Dive
50
• History of Logging
– Originally was only synchronous, but
this didn’t scale so from Mule 3.6+
libraries were changed to use
asynchronous logging
– Uses the Log4j framework which has
progressed onto log4j2
– Very limited slideshare or articles
about mule logging publically available
– Exam questions are limited to types of
logging produced by connectors and
the runtime upon startup or which logs
get called depending on flow logic

51
• Asynchronous Logging
– A thread created for each
log4j2.xml in started apps,
domains and mule servers
– Threads are flushed on server
shut down, but logging events
can be lost on system crash
– Logging threads fail over to
server log, ie mule_ee.log
Synchronous Logging
– Can be used in conjunction
with asynchronous logging
– Performance impacted slightly

52
• Logging Bad Practice
– No Logging standards
• mix of json, text or empty loggers, information
not complete
– Hard Coding of loggers
• Not able to adjust via environment variables or
test for error content
– Empty Loggers
• These can fill logs and confuse debugging
– Logging Limits not respected;
• Cloudhub is limited to 100Mb or 30 days
– Sensitive Information
• Usernames and passwords should be reported in
encrypted or obfuscated format

53
• Logging Best Practice
– Create standards
• Use json with key: value pairs
• Form common error and logging structures
• Extract Flows to external Dependencies
– Informative Messages
• Visibility into the system (where log is raised)
• Enhanced debugging available (dynamic error
levels)
• Before and after system call logs (with payload)
– Request Identifier
• Use the correlationId or assigned code for
transaction (this can help with testing latency)
– Persist Logs
• Maintain historical logs
• Allow non-technical users to access and support
applications

54
• Three types of logging
– Runtime
– Connectors
– Custom (Logger element)
Hierarchy of Configuration Files
– Application Resources
– Mulesoft Server conf
– Default (see image)
Override in Deployment
- Using deployment properties (3.8+)
- Configuration Reloading
- Disabling Logging

55
• Disabling Cloudhub Logging
– Mulesoft Documentation Article
– Changing Cloudhub directly
requires raising support tickets
– This configuration allows the
local log4j configuration to be
used in place of the default
cloudhub configuration

56
• Disabling Complete Logging
– Mulesoft Knowledge Article
– Appenders element will redirect
all logging to console
– The Logging elements sets the
level to OFF, preventing any
output being written to console
– Can be used to rule out logging
issues in performance testing

57
• Live Logfile Updating
– Not configured in the default or
template configurations
– Can be used to make logging
more or less verbose in live or
nearly live environments
– Would require access
permissions to the mule server
to edit the log4j2.xml file

58
• Different Log Files
– Additional Thread is created
for the separate
AsyncLogger
– Can be used to separate
business logic and standard
INFO logs from error and
debug logs
– Debug logs can include
connection strings and
sensitive information

59
• Configuring Custom Log
– Edit ‘mule-deploy.properties’ file
located in app folder
– Can be used to define default
logging file
– Limited by no env variable for
appname
– May come into its own with
Runtime Fabric
– Gets around needing access to
mule_home

60
• Verbose Logging
– Module & Transport names
(list to the right)
– Levels are as per SLF4J
specification (see below)
Amazon S3 org.mule.module.s3
Amazon SQS org.mule.modules.sqs
Anypoint MQ
org.mule.module.messaging.mq
com.mulesoft.mq
okhttp3
okio
DB org.mule.module.db
Email org.mule.transport.email
File org.mule.transport.file
FTP
org.mule.transport.ftp
com.mulesoft.mule.transport.ftp
JMS org.mule.transport.jms
Microsoft Service Bus org.mule.modules.microsoftservicebus
Mongo org.mule.module.mongo
Salesforce
org.mule.modules.salesforce
com.sforce
org.mule.module.http.internal.HttpMessageLogger
SFTP
org.mule.transport.sftp
com.jcraft
Object store org.mule.modules.objectstore
TCP org.mule.transport.tcp
UDP org.mule.transport.udp
VM Transport org.mule.transport.vm
WMQ com.mulesoft.mule.transport.wmq

61
• Separate Logging Levels
– Create sub flows with multiple
log elements
– Use elements for standard
business logic and separate
debug including payload
– Exception handling could also
include similar multiple log
elements
– Can be switched to info/debug

62
• Recommendations
– Separate custom log messages
based on env injected
properties
– Create custom classes of log
messages for reporting
business logic
– Capture a standardized
logging format for exceptions,
business logic, connector
information and validations
– Look at using RFC 7807 to
format error messages

63
• Further Research
– Mulesoft Blog
• There was a useful blog about logging
from a Mulesoft trainer in October
2019
– Meetup Example Flows
• The Denver meetup group shared a
project with useful examples of bad
and best practice
– Custom Connector Code
• BigCompass have shared a custom
connector built using the Mulesoft
DevKit to formalise logging structure

Does this approach to project delivery look familiar?

Enable devops: establish your delivery pipeline
ProductionNon-Prod
Agile Product Delivery Management
Feedback
Service
Management
3rd-party
Log Analytics
API
Portal
API
Designer
Anypoint
Studio
Anypoint
Exchange
Anypoint Runtime Manager
Anypoint Monitoring
Ricston webinar: Deliver Mule Applications Faster with CI/CD
Refine Discover Engage Feedback

Enable devops: establish your delivery pipeline
ProductionNon-Prod
Agile Product Delivery Management
Feedback
Service
Management
3rd-party
Log Analytics
API
Portal
API
Designer
Anypoint
Studio
Anypoint
Exchange
Anypoint Runtime Manager
Anypoint Monitoring
Ricston webinar: Deliver Mule Applications Faster with CI/CD
Refine Discover Engage Feedback
Version
Control
CI/CD Pipeline ChatOps
Artifact
Repository

Be API led: establish an API design function
By Pieter Bruegel the Elder - The Tower of Babel
(Google Cultural Institute):
https://commons.wikimedia.org/w/index.php?curid=22
178101

Reuse >
Enhance > Create
Balance
Simplicity vs Flexibility
Domain Driven
Data Architecture
Blend API evangelism, design and delivery experience to productise digital
capabilities as APIs aligned to promise and strategy of the business

Establish common Mule development standard
● Adopt source control and select prefered development methodology
○ Branch vs Trunk based development
● Define Mule development conventions:
○ Names: files, configuration, classes, properties
○ Structure: project layout
○ Scope: header, variable and payload usage
○ Property management: environments, secure properties
○ Test approach and coverage
○ Exception handling
● Refactor conventions and guidelines into reusable templates or libraries
○ Bake “right-fit” governance in; make it easy for developers to adopt
● Automate compliance checks

Establish common Mule development standard

{
"type": "validation_failure"
"message": "request data is not valid"
"details": [
{
"field": "customer.address[2].type",
"code": "unexpected_value",
"reason": "'WORK' is not a defined address type: HOME, BILLING,
SHIPPING, BUSINESS, RECIPIENT",
"moreInfo": "http://api.example.com/docs/customer#address_types"
},
{
"field": "customer.address[2].postcode",
"code": "invalid_value",
"reason": "'N19 X6Z' is not a valid postcode"
}
]
}
● Define API error response
○ Error types != HTTP status code
○ Do not expose implementation specifics
○ API docs should maintain a glossary of API error types
● Consider business validation scenarios
● Define organisation (or domain) specific RAML Library
● Extend MuleSoft’s default global exception strategy
○ Consider API resource specific overrides
● Log exception details and useful context
○ ERROR or WARN log levels exist for a reason!
Standardise API error behaviour

{
"details": [
{
},
{
}
]
}

{
"details": [
{
},
{
}
]
}
● Define API error response
○ Error types != HTTP status code
○ Do not expose implementation specifics
○ API docs should maintain a glossary of API error types
● Consider business validation scenarios
● Define organisation (or domain) specific RAML Library
● Extend MuleSoft’s default global exception strategy
○ Consider API resource specific overrides
● Log exception details and useful context
○ ERROR or WARN log levels exist for a reason!
Check out: RFC7807 - Problem Details for HTTP APIs

Standardise what and how your APIs log
● Translate obscure organisation policies into tangible developer
guidelines
○ Consider role of log levels
● Identify core set of attributes to provide API context and front-to-back
traceability
○ Unique identifiers: client, request, conversation
○ Method, resource, headers, parameters
● Define a log event format (K:V or JSON)
● Define log event generation mechanism
○ Mule logger component
○ Custom message processor
○ Log4j appender/layout
● Log analytics: create dashboards that enable more proactive monitoring
○ Ability to drill into client (4xx) and server (5xx) errors

Key Takeaways
Simplify delivery through
consistency that removes
friction from development and
operations
Digital capabilities designed
and operated as products, to
drive competitive advantage &
business value
Automate repeatable
processes that increase quality,
feedback and accelerate
delivery cycle
Invest in the foundations to generate platform-wide returns

Thank you! What’s next?
● Feedback forms
● Next meetup - end January / beg February
● Interested in presenting? Any topics to
suggest? Let us know!
● Now - networking & drinks
Manchester, UK | 29 October, 2019
MuleSoft Inc.I www.mulesoft.com
Ricston Ltd. | www.ricston.com

MuleSoft Manchester Meetup #2 slides 29th October 2019

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a MuleSoft Manchester Meetup #2 slides 29th October 2019

Semelhante a MuleSoft Manchester Meetup #2 slides 29th October 2019 (20)

Mais de Ieva Navickaite

Mais de Ieva Navickaite (6)

Último

Último (20)

MuleSoft Manchester Meetup #2 slides 29th October 2019