bbcon 2013

1. 10/7/2013 #bbcon 1
BBCRM Performance
From Cobblestones to Autobahnen
PRESENTED BY LEN WYATT

2. 10/7/2013 #bbcon 3
Cobblestones? Autobahnen?
Why such a silly subtitle?

3. 10/7/2013 #bbcon 4
WHAT IS NEEDED TO BUILD A
COBBLESTONE ROAD

4. 10/7/2013 #bbcon 5
WHAT IS NEEDED TO BUILD AN AUTOBAHN

5. 10/7/2013 #bbcon 6
WHAT IS NEEDED TO USE A
COBBLESTONE ROAD

6. 10/7/2013 #bbcon 7
WHAT IS NEEDED TO USE AN AUTOBAHN

7. 10/7/2013 #bbcon 8
• To schnell fahren auf der Autobahn, there is a lot behind the scenes!
• The same is true for complex multi-user systems like BBCRM
• So when something goes wrong, why is software so much harder to
troubleshoot?
LAYERS OF INFRATRUCTURE,
KNOWLEDGE AND SUPPORT

8. 10/7/2013 #bbcon 9
• Browser
• Operating System
• Client computer
• CPU, Memory, Disk
• Local LAN
• Internet
• Web server
• CPU, Memory, Disk
• Data center LAN
• Database server
• CPU, Memory, Disk
• Database schema
• Database contents
• Stored procedures
• Customizations
LAYERS OF ABSTRACTION
What the user did
What everybody else is doing
What is stored in the systems
How the systems have been configured
How the systems have been maintained
Developers learn to use these abstractions
to hide the behavior of other layers
Performance issues can occur in any layer.
Everything is subject to examination.

9. 10/7/2013 #bbcon 10
• Because abstraction is so easy in software, we do it a lot!
• All the tools, infrastructure and support are built on a
digital background that we can’t physically see – we have
to hold all the concepts in mind
• Evolution has not prepared us for this
• We do not have hundreds of years of shared experience
LAYERS OF ABSTRACTION

10. 10/7/2013 #bbcon 11
• “It’s taking a long time” does not point to the problem
• This is not the user’s fault
- The layers of abstraction hide the issue
• We have to ask a lot of questions, and use tools to collect information
WHAT THE USER CAN TELL US

11. 10/7/2013 #bbcon 12
ENTER THE ENTERPRISE PERFORMANCE TEAM

12. 10/7/2013 #bbcon 13
ENTERPRISE PERFORMANCE TEAM
WORK AREAS
• Tactical improvements. Some customers have
performance issues today, and we need to
address key ones. Tactical work focuses on
carefully chosen areas that are common to
multiple customers, and seeks solutions.
Recommendations are made to product teams.
• Strategic improvements. Long term, we must
find ways to improve the product design and
engineering processes to focus on performance
early and continually.
• Practices. Deliver information to help
customers as well as our own customer-facing
teams (SDO, Prof. Svcs, Support, Sales) do
their jobs better.

13. 10/7/2013 #bbcon 14
THIS IS A NEW APPROACH
• Yep, we have had some
performance problems with BBCRM
• No, we haven’t always handled them well
• It feels like we have been playing
Whack-a-Mole
• Now we have…
• A focused team
• Working on root causes
• Being scientific
• Bringing new thinking about
performance to the organization

14. 10/7/2013 #bbcon 15
Performance work across the organization…
Product performance deliverables:
• Ad-Hoc Query Improvements
• Revenue Batch Improvements
• Security Model Improvements
• Blackbaud Direct Marketing
segmentation on the DW
Driven by development teams
• Planned, scheduled work
BBCRM PERFORMANCE INITIATIVES
Research-based approaches:
• Tactical improvements
• Strategic improvements
• Practices information
• All based on representative
workloads
Driven by the
Enterprise Performance Team
• Exploratory & opportunistic work

15. 10/7/2013 #bbcon 16
ONE TACTICAL IMPROVEMENT
• Individual Revenue Summary View Data Form
• Found issue by observing Slow Web Requests in SDM
• Issued patch for 2.93 (later versions were already improved)
• The impact on AHA alone:
• Before (6/1/2013 – 6/30/2013)
• 4,889 SWRs
• Average 60 seconds per SWR (55 seconds over threshold)
• After (7/1/2013 – 8/5/2013)
• 113 SWRs
• Average of 7 seconds per SWR (2 seconds over threshold)
• Nearly ½ of a full time person no longer spent waiting!

16. 10/7/2013 #bbcon 17
• Postings on the BBCRM blog
- http://www.blackbaudknowhow.com/blog/blackbaud-crm
EARLY PRACTICES RESULTS

17. 10/7/2013 #bbcon 18
• Who has seen those blog postings?
• What is the most effective way to get information out to you?
• Is it different depending on the type of information?
- User tips
- Database administrator recommendations
- Configuration & deployment guidance
QUESTIONS!

18. 10/7/2013 #bbcon 19
WE WORK BY INVESTIGATIONWE WORK BY INVESTIGATION: WE DO SCIENCE

19. 10/7/2013 #bbcon 20
• We study and learn from customer experience
- Review and analyze Work Items related to performance
- Study available instrumentation
• SDM data for hosted customers
• We rarely work directly on a specific customer case
- Because we are looking for the root causes that affect
many customers
• We examine the workings of a system in operation
- To see what’s really happening, and not speculate
• We work with engineering teams to implement changes
- Code ownership stays where it belongs
- Developers and QA people learn from the process
SCIENCE STARTS WITH OBSERVATION

20. 10/7/2013 #bbcon 21
• To study a working system, a testbed is required
- Does it help if we make this change?
- Are there side-effects from changing this?
- Is alternative A better than B?
- Is this configuration optimal?
- What if we have more (of fewer) CPUs? Memory? Disks?
- Could we support more users?
- What is the source of database growth?
- How does index fragmentation change performance?
- Would database compression help performance?
- … and on and on…
• Using the testbed requires a workload
- Models the activities the system is supporting
- Models the data in the system
- Allows measurement of response times
TESTBED

21. 10/7/2013 #bbcon 22
• Both the state of the SUT and the applied work must be meaningful in
order to get a meaningful result
- Mostly this means the database contents and the user actions
• Representative workloads will allow us to do science instead of guessing
MEANINGFUL WORKLOADS ARE VITAL
FOR PERFORMANCE WORK
Database
Reflects the rich
complexity of real
customer
databases
Web Tier
User Actions
Reflect what
users do and
action rates
Test
Driver
System Under Test

22. 10/7/2013 #bbcon 23
• Workloads model user actions
- What do users do?
- What parameters are filled in?
- How much time between actions?
- How many users?
• Workloads model the database contents
- How many constituents? Addresses? Contributions? Reports? Tributes?
- The patterns in the data are important…
• Do some constituents make many contributions and others only a few?
• Are there some for whom we have lots of address changes recorded?
• What is the ratio of general contributions vs. directed ones?
• Creating a workload like this is expensive; we can’t do it too often!
- So it has to be representative of many customers
WORKLOADS

23. 10/7/2013 #bbcon 24
• To create lab workloads that are highly
representative of real customer workloads,
we are modeling from systems run by
Performance Partners
- If we do our job well, the lab workloads will be
almost doppelgangers of the originals
• Performance Partners…
- Are BBCRM customers
- Chosen because they have issues representative of multiple others
- Agree to work with us
- Let us work with a (masked) copy of their database
- Let us monitor user activity for an extended time (a month)
- Agree to let us instrument systems from time to time, if needed
- Want to try updates when applicable
- Have willingness and capability to test, measure and try some things
PERFORMANCE PARTNERS

24. 10/7/2013 #bbcon 25
• Heifer International
• The University of Iowa Foundation
• They are different…
- Usage models
- Sizes of databases
- Patterns of performance issues
- Hosted by Blackbaud vs. self-hosted
• They are similar to others in their categories
PERFORMANCE PARTNERS

25. 10/7/2013 #bbcon 26
• What system configurations do our customers use?
- For hosted customers, we know
- For self-hosted customers, we have only spotty information
• How can we know if our testbed is representative of self-hosted sites?
• Help us help you…
- We have sent a survey through the TAMs
- It asks about physical equipment
- Accompanied by a script that collects
statistics on the database
• If you haven’t seen this, ask your TAM
- Have them ask me if they don’t know 
TESTBED, REDUX

26. 10/7/2013 #bbcon 27
THE SELF-HELP SECTION

27. 10/7/2013 #bbcon 28
• End Users
- Watch for tips like we have posted on the BBCRM blog
• We have posted about Constituent Search and Querying by Merging
Selections
- Try not to run large batch operations or slow reports during prime time
• System administrators
- Make sure maintenance tasks are performed regularly
• Update statistics
• Rebuild indexes
- Eliminate unnecessary data
• Excessively old audit data
• Unused export tables
- Know your baseline performance
• Monitor PerfMon counters
MAKING THINGS BETTER YOURSELF

28. 10/7/2013 #bbcon 29
• Developers
- Familiarize yourselves with SQL query tuning techniques
- Follow known best practices
• Learn them at talks tomorrow: “Blackbaud CRM Boot Camp” Parts 1 & 2
• Support staff
- Be familiar with what Blackbaud support will need to know about
performance issues
• You can start to ask the same questions
MAKING THINGS BETTER YOURSELF

29. 10/7/2013 #bbcon 30
HELP BB SUPPORT HELP YOU!

30. 10/7/2013 #bbcon 31
• Was any of that helping-yourself material a surprise?
• Which of those things are you doing today?
• What tools do you use for routine maintenance tasks?
DISCUSSION

31. 10/7/2013 #bbcon 32
FIN
• We’re taking a whole new approach to performance
• This is going to take time
• The benefits are going to be big
• We are improving BBCRM
• We are changing the organization

32. 10/7/2013 #bbcon 33
WHEW! THAT WAS A LOT OF MATERIAL.