Optimizing BbLearn 9.1 for High Performance

1. Best Practices for Optimizing
Blackboard Learn
Steve Feldman,
sfeldman@blackboard.com

2. optimizing* The process of increasing speed
(performance) and capacity (scalability) of a system
and/or program.

4. What We’ll Cover
• Deploy for performance from the start.
• Optimizing the platform components.
• Continuous measurements are absolutely critical.

5. Scalable Deployments

6. Flexible and Scalable Application
Deployment

7. Deployment: Large Address Space
• As of Blackboard Learn™ Release 9.1 all supported/
certified configurations include a 64-bit option.
• Pushing more processing to client and DB over the last
few releases, but major memory management technique
is to use more application caches.
– Memory stays persistent longer
– Less wasteful from a creation/destruction perspective, but puts
greater demands on larger spaces.
• Most of our application testing focused on 4GB and 8GB
JVM deployments on 6GB and 10GB OS spaces.
– Limited testing at 16GB and 32GB

8. Deployment: Resource Utilization via Virtualization
• Moore’s law is in full effect
– CPUs are getting faster with more cores
– Memory is in abundance and cheap
– Storage is grossly abundant
• Massive systems can be obtained at low cost, but cannot
be saturated in stand-alone configurations.
• Virtualization offers this opportunity…
– Deploy with availability in mind
– Saturate system resources
– Save money and data center space

10. Deployment: Improving Page Responsiveness
• Gzip…Gzip…Gzip…
– All of our supported browsers handle gzip?
– Reduces payload
• Improves lower latency connections like Cable, DSL and Dial-up
– Minor overhead on the application layer (~2% to ~5%)
• Have the option to perform at the load-balancer layer
– Most Bb deployments do not enable Gzip at all
• Even when enabled, some proxies and software packages mess-up
the Accept Encoding Header
• Optimize your images
– Page size really does matter
– Reduce the size without reducing the quality
– Smush.it, PunnyPng, OptiPng, RIOT and ImageOptim

11. Example Page Waterfall

13. Tools for Page Performance
• Analysis Tools
– Independent tools: Fiddler, HTTPWatch, WebPageTest.org, Web
Inspector (Mac), IE Developer Tools
– Browser Plugins: Firebug/YSlow, Google Page Speed,
WebPageTest Client
– JavaScript tools: dynatrace, IE Developer Tools,
• Optimization Tools
– Image optimization to reduced overall size and metadata:
SmushIt, PNGOptimize, PNGCrusher, OptiPng and
graphicsoptimization.com
• http://yuiblog.com/blog/2008/10/29/imageopt-1/
– CSS optimization to reduce unused elements and compress
CSS of unnecessary white spaces: CSS Optimizer, CSS
Compressor and CleanCSS

15. Optimizing the Web Server
• The web server in the BbLearn configuration is nothing
more than a gateway to the application container.
– When clusters were more relevant, the web server acted as a
pseudo load-balancer.
• Not many opportunities for optimization other than
– KeepAlives
– Interfaces
– Compression
• It can become a bottleneck if not properly optimized
– Better to have high ceilings from an interface perspective

16. Optimizing the JVM
• Java hotspot offers standard –X and non-standard –XX
options for performance and behavior.
– -X options are always guaranteed across releases and patches of
Java.
– -XX options must be used with caution as they are subject to
change with any release of Java.
• -XX options should be tested and measured using the
production safe arguments.
– Be careful of leveraging options you see on ListServes and
Google
• Read the release notes of Java for “performance” updates
– http://java.sun.com/javase/6/webnotes/ReleaseNotes.html

17. Regions of the JVM

18. Optimizing the JVM
• Cross-platform recommendation for using Concurrent
Mark Sweep Collector
– Best optimized for 64-bit address
– Combine –XX:+UseConcMarkSweepGCwith –XX:+UseParNewGC
• Manually size New Space using –XX:NewSize and –
XX:MaxNewSize options (1/4 to 1/3 total heap).
– Consider Survivor Space ratios 4 or lower.
• Be careful about sharing –XX non-standard options across
customers.
– If you don’t understand what the option does and it’s not
recommended by Blackboard, best choice is to not use it.
• PermSpace and B2s
– Every B2 you add, make sure to measure PermSpace usage via
tools like JStat and VerboseGC

19. High Level Old Space Collection

20. Prototyping JVM Arguments
• Wrapper.conf exists on both
UNIX and Windows
configurations for handling
JVM arguments.
• Options can be tested out
without a
PushConfigUpdates by
simply adding an additional
line with
– wrapper.java.additional.<insert
numeric sequence>=
– Simply issue a “./
ServiceController
services.appserver.restart” to
restart the JVM without
arguments

21. PrintGCStats

22. Tools for Java Instrumentation
• GC Log Analysis Tools: PrintGCStats, Jstat,
GCViewer,
• Thread Analysis Tools: Samurai, JConsole,
JavaVisualVM,
• Memory Analysis Tools: Eclipse Memory Analyzer,
Jhat,
• Robust Commercial Tools: Quest Foglight, Oracle
OEM, Dynatrace for Java, CA IntroScope
• Profiling Tools: Jprobe, YourKit, JProfiler

23. Optimizing the Database: SQL Server
• # of data files makes no difference on SQL Server for
Data and Transaction
• Allow the data/transaction files to grow as big as they
want within reason.
– What’s reason: 64GB
– http://msdn.microsoft.com/en-us/library/ms143432(sql.90).aspx
• TempDB is completely different story
– # of files = # of DB Threads
– Set first X files to a uniform size, set last file to same size with
auto-extension ON
– Determine size need over time
• Separate volume for paging file

24. Optimizing the Database: SQL
Server
• Be aware of MDOP: Max Degree of Parallelism
– Setting to unlimited can have a negative affect on query
performance unintentionally.
• AWE can and does work on 64-bit systems
• Configure READ_COMMITTED_SNAPSHOT
• Two nuggets of information:
– Learn How to Use SQL DMVs
– Study SQL Server Wait Events and Tuning

25. Optimizing the Database: Oracle
• Balance I/Os across multiple data files (~2 to 8GB per
file).
• REDO is critical to performance a session/query level.
– Be aware of how much REDO is being used over time.
– NOLOGGING will disable, be we rarely use NOLOGGING
• TEMP is very complex and used for managing transient
data.
– One TEMP file is adequate
– If latency exists on TEMP, consider introducing TEMP file groups
• SGA is important, but PGA can be your best friend or your
worst enemy with high concurrency.

26. Optimizing the Database: Oracle
• Oracle CBO can be your friend
– Must understand optimizer behavior
– Willingness to read Cost Execution Plans
• Using Wait Events and Cost Execution Plans for tuning
initiatives
– Wait events are at a system, session and query level
• Importance of Statistics and Histograms
– CBO is just guessing without properly set statistics and
histograms.
– CBO is dependent on your data.

27. Approach to Understanding DB Performance
• Capture statements using some pre-identified tool
– Oracle AWR and ASH (Session Level)
– SQL Server Performance Dashboard and Performance
Warehouse
• Review Instance Level Information to gain a cursory view
of “system” performance.
• Deep dive into SQL Analysis
– Focus on Logical I/Os as starting point.
• Physical I/Os are important, but more often logical
– Drill into CPU consumers followed by executions
– Isolate physical and structural for over usage and possible
locking conditions
– Question why executions, but 0 rows processed
– Importance of both: Histograms and Statistics
• Know your data better than your DBAs

29. Please provide feedback for this session by emailing
BbWorldFeedback@blackboard.com.
The subject of the email should be title of this
session:
Best Practices for Optimizing Blackboard Learn