Human Factors of XR: Using Human Factors to Design XR Systems
Clustered Architecture Patterns for Scalability and Availability
1. Clustered Architecture Patterns:
Delivering Scalability and Availability
Orion Letizi – Terracotta
Co-Founder and Software
Engineer
Confidential – for information of designated recipient only. Copyright Terracotta 2006
2. Agenda
Patterns from Years of Tier-Based Computing
Network Attached Memory / JVM-level clustering
Clustered Architecture Patterns
In-depth Review Of The New “Examinator” Reference
Implementation
Confidential – for information of designated recipient only. Copyright Terracotta 2006
3. The Innocent Enterprise Application
Within every innocent enterprise application lies a sleeping
monster
There comes a time when every successful enterprise application
outgrows its single machine architecture
Whether for high-availability or scalability or both, the adult
enterprise application must live on more than one application
server
That's when the latent beast strikes...
Confidential – for information of designated recipient only. Copyright Terracotta 2006
5. The State Monster
User-scoped session state
– Conversational state across
requests
Application state
– Global data, caches, indexes, etc.
Confidential – for information of designated recipient only. Copyright Terracotta 2006
6. The State Monster
At Walmart.com we started like everyone else: stateless + load-balanced
+ Oracle (24 cpus, 24GB)
Grew up through distributed caching + partitioning + write behind
We realized that “ilities” conflict
– Scalability: avoid bottlenecks
– Availability: write to disk ( and I/O bottleneck )
– Simplicity: No copy-on-read / copy-on-write semantics (relentless tuning, bug fixing)
And yet we needed a stateless runtime for safe operation
– Start / stop any node regardless of workload
– Cluster-wide reboot needed to be quick; could not wait for caches to warm
The “ilities” clearly get affected by architecture direction and the stateless
model leads us down a precarious path
Confidential – for information of designated recipient only. Copyright Terracotta 2006
7. The Precarious Path: Our tools lead us astray
Stateless load-balanced architecture ⇒ bottleneck on DB
In-memory session replication ⇒ bottleneck on CPU, Memory
Clustered DB cache ⇒ bottleneck on Memory, DB
Memcache ⇒ bottleneck on server
JMS-based replicated cache ⇒ bottleneck on network
…Pushing the problem between our app tier CPU and the data
tier I/O
Confidential – for information of designated recipient only. Copyright Terracotta 2006
8. CRUD Piles Up…
Types of clustering:
– Load-balanced (non-partitioned) Scale Out
– Partitioned Scale Out
Both Trade-off Scalability or availability (usually by hand) in
different ways
scalability
availability
…and everything we do forces the trade-offs
Confidential – for information of designated recipient only. Copyright Terracotta 2006
9. The Stateless Convention: Use The Database
Confidential – for information of designated recipient only. Copyright Terracotta 2006
10. The Stateless Convention: Push State to the Database
DB becomes a bottleneck
Not optimized for object data
Confidential – for information of designated recipient only. Copyright Terracotta 2006
11. The Stateless Convention: Push State to Peers
Confidential – for information of designated recipient only. Copyright Terracotta 2006
12. The Stateless Convention: Push State to Peers
Push state everywhere gives HA, but doesn't scale
Push conversational state to buddy can't survive buddy pair death or
full cluster restart. Doesn't help for application state.
Confidential – for information of designated recipient only. Copyright Terracotta 2006
13. The Stateless Convention: Push State to Client
Highly scalable for conversation state UNLESS state is large
Doesn't help with application state
Confidential – for information of designated recipient only. Copyright Terracotta 2006
14. The Stateless Convention: Eats Developer Brains
Confidential – for information of designated recipient only. Copyright Terracotta 2006
15. The Stateless Convention: Eats Developer Brains
Pushing state out of app server
doesn't get rid of state. There
is no such thing as quot;statelessquot;
Serialization/externalization:
– Full-object graph serialization
– breaks object identity
– corrupts data and programming
model
quot;with a little ingenuity...quot;--
seems fun/easy to try, but ends
up taking over the development
effort
Confidential – for information of designated recipient only. Copyright Terracotta 2006
16. Changing the Assumptions: JVM-level Clustering
Confidential – for information of designated recipient only. Copyright Terracotta 2006
17. What Terracotta Is
Definition
From “The Definitive Guide…”
– Terracotta is Java infrastructure software that allows you to scale your
application for use on as many computers as needed, without expensive
custom code or databases
NAS but for memory (almost)
– A server process for central storage and locking
– A transparent client I/O driver
– A network protocol
Network Attached Memory (NAM)
Confidential – for information of designated recipient only. Copyright Terracotta 2006
18. What Terracotta Is
Terracotta Deployment Model
Your JVMs
1. Uses Hotspot JVM
2. TC libraries are jars and bootclassloader
3. Terracotta Server is pure Java
TC Libraries 4. Terracotta Stores all data to disk
5. Clustering in the box
TCP/IP
Terracotta Terracotta
Server Server
JVM JVM
replication
disk disk
Confidential – for information of designated recipient only. Copyright Terracotta 2006
19. What Terracotta Is
Changes In 1 JVM Visible in Others
JVM JVM JVM
Terracotta
Confidential – for information of designated recipient only. Copyright Terracotta 2006
20. What Terracotta Is
…Like This
JVM JVM JVM
mutate ACK ACK
Establish
single update
Persistent
TCP broadcast
Connection
Terracotta
Confidential – for information of designated recipient only. Copyright Terracotta 2006
21. What Terracotta Is
Oh Yeah: Did I Mention It is JVM-Level?
Before Terracotta After Terracotta
ID Monitor
ID Monitor
Object Object
Primitive Method() Primitive Method()
Field (value) Field (value)
Object Object
Field (ref) Field (ref)
ID Monitor ID Monitor
Confidential – for information of designated recipient only. Copyright Terracotta 2006
22. Performance + Reliability + Simplicity
10X throughput over conventional APIs
– All Reads from memory (implicit locality)
– All Writes are deltas-only
– Statistics and heuristics (greedy locks)
Terracotta Active Server Array
– Simple form available today for enterprise customers
– V1.0 GA this year
Looks like Java to me (code like your mom used to make)
– Normal access patterns: no check-out before view and check-in on commit
– Code and test at a unit level without infrastructure intruding on app logic
– Threads on multiple JVMs look like threads on the same JVM
Confidential – for information of designated recipient only. Copyright Terracotta 2006
23. What Terracotta Is
Batching and Ordering In Accordance With JMM
Durable
Transactional
Clustered
Shared Memory
Use synchronized or util.concurrent or Terracotta Integration
Module to lock
Confidential – for information of designated recipient only. Copyright Terracotta 2006
24. What Terracotta Is
What TC Does for You
Pure Java
– Shares your objects and framework internals
– Shares thread coordination
Efficient Scale-out through heuristics and statistics (what is
resident where)
Availability in the box
Confidential – for information of designated recipient only. Copyright Terracotta 2006
25. What Terracotta Is
Reduces Clustering to Tuning
Always coherent
Always on disk at in-memory speed (streaming, not seeking)
Always Pure Java (java.*, synchronized, and util.concurrent)
Changes needed usually for tuning
Confidential – for information of designated recipient only. Copyright Terracotta 2006
26. When to use Terracotta: Will It Work With My App?
(The Terracotta API, so to
speak)
Confidential – for information of designated recipient only. Copyright Terracotta 2006
27. When to Use Terracotta
Users and Use Cases
Web Apps Customer Service
– Portals – CRM apps
– Social networking – Rewards / affinity programs
– Gaming, betting
Financials Telco / IT
– Trading apps – VOIP
– Trading bus – Server mgmt
Confidential – for information of designated recipient only. Copyright Terracotta 2006
28. When to Use Terracotta
Terracotta is a Platform (not an app server)
How do you share objects
– HashMap, CHM, EHCache, HashTable, TreeMap
How do you coordinate access
– Wait / Notify, synchronized, util.concurrent
Where do you store the data
– In Terracotta as objects, write-behind to DB, write-thru to DB, etc.
How do you process async workloads
– Queue of changes, eventing pattern, etc.
Confidential – for information of designated recipient only. Copyright Terracotta 2006
29. When to Use Terracotta
API: Using Patterns That Are Clusterable
Conversation clustering / persistence
Window onto large dataset
Data as a service
Cluster membership and management
Change Notification
Shared Metadata
Confidential – for information of designated recipient only. Copyright Terracotta 2006
30. When to Use Terracotta
Pattern: Conversation Clustering
Set a conversation key (cookie)
Value should be object graph
– not primitives like string or byte array
Load Balancer required
– Sticky load balancing (layer 7 preferred,
or Apache mod_jk)
– Ensures locality of reference
No DB required
Ref. impl. code available now:
http://svn.terracotta.org/svn/forge/projects/exam/
Confidential – for information of designated recipient only. Copyright Terracotta 2006
31. Account Creation
SignupController
– UserValidator
– RegistrationService (DefaultRegistrationService)
@RequestMapping(method = RequestMethod.POST)
public ModelAndView processSubmit(final HttpServletRequest request, @ModelAttribute final User user,
final BindingResult result, final SessionStatus status)
throws RegistrationException, ProtocolException {
validator.validate(user, result);
if (result.hasErrors()) {
return new ModelAndView(quot;registration/signupquot;);
} else {
// make sure that no clear text passwords are stored in the back-end
user.setAndEncodePassword(user.getPassword());
// hold the user's registration until it has been confirmed over email
this.service.holdForEmailConfirmation(user, UrlHelper.createAbsoluteUrl(request,
ConfirmController.class));
status.setComplete();
return new ModelAndView(quot;registration/successquot;, quot;userquot;, user);
}
}
Confidential – for information of designated recipient only. Copyright Terracotta 2006
32. Account Creation: RegistrationService
DefaultRegistrationService
public class DefaultRegistrationService implements RegistrationService {
private final Configuration freemarkerConfiguration;
private final UserService userService;
private final MailSender mailSender;
@Root
private final Map<String, Long> heldUsers = new ConcurrentHashMap<String, Long>();
//...
public void holdForEmailConfirmation(final User user, final String confirmationUrl)
throws RegistrationException {
// ... Validate args
final String uuidString = SecurityHelper.generateUniqueCode();
heldUsers.put(uuidString, user.getId());
// ... Code to send confirmation email
}
}
Confidential – for information of designated recipient only. Copyright Terracotta 2006
33. Account Creation
Durable Java data structure as temporary storage for confirmation
codes
Benefits
– No round trip to the database
– No need for caching
– No cache freshness issues (no opportunity for arbitrage)
– No impedance mismatch at O/R layer
Issues
– Must clean up data structure in a cluster-aware way (similar to HTTP Session
cleanup)
Password reset function works the same way
Confidential – for information of designated recipient only. Copyright Terracotta 2006
34. AuthC/AuthZ
Spring Security TIM
– Authentication (AbstractAuthenticationToken) and authorization
(GrantedAuthorityImpl) tokens held in HTTP Session via Spring Security
<filter>
<filter-name>springSecurityFilterChain</filter-name>
<filter-class>org.springframework.web.filter.DelegatingFilterProxy</filter-class>
</filter>
<filter-mapping>
<filter-name>springSecurityFilterChain</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>
– Clustered via Terracotta session clustering module
– Spring Security TIM handles all instrumentation configuration for objects in
HTTP session
LoginController
Confidential – for information of designated recipient only. Copyright Terracotta 2006
36. AuthC/AuthZ: LoginController
@Controller
@RequestMapping(quot;/login.doquot;)
public class LoginController {
@RequestMapping(method = RequestMethod.GET)
public ModelAndView loginHandler(final HttpServletRequest request,
final HttpServletResponse response, final HttpSession session) {
final ModelAndView result = new ModelAndView(quot;user/loginquot;);
final boolean loginError = request.getParameter(quot;login_errorquot;) != null;
if (loginError) {
result.addObject(quot;loginErrorquot;, loginError);
final String lastUser
= (String) session.getAttribute(AuthenticationProcessingFilter.SPRING_SECURITY_LAST_USERNAME_KEY);
final Throwable lastException
= (Throwable)session.getAttribute(AbstractProcessingFilter.SPRING_SECURITY_LAST_EXCEPTION_KEY);
if (lastUser != null) {
result.addObject(quot;lastUserquot;, lastUser);
}
if (lastException != null) {
result.addObject(quot;errorMessagequot;, lastException.getMessage());
}
}
return result;
}
}
Confidential – for information of designated recipient only. Copyright Terracotta 2006
37. Conversation State: Take The Exam
Spring WebFlow
– Workflow through examination process
ExamService (CachingWrapperExamService)
– Caches exam meta-data (questions, answers, sections, etc.)
ExamSessionService
– Manages the state of in-progress examinations using clustered:
– ReentrantReadWriteLock
– Map
Confidential – for information of designated recipient only. Copyright Terracotta 2006
38. Conversational State: List Available Exams
<view-state id=quot;chooseExamquot; view=quot;exam/listquot;>
<on-render>
<evaluate expression=quot;examService.getAllExams()quot; result=quot;viewScope.examsquot; />
</on-render>
<transition on=quot;selectExamquot; to=quot;examSelectedquot;>
<set name=quot;flowScope.examIdquot; value=quot;requestParameters.examIdquot;></set>
</transition>
<transition on=quot;returnquot; to=quot;returnquot; />
</view-state>
Confidential – for information of designated recipient only. Copyright Terracotta 2006
39. this.cacheManager = CacheManager.getInstance();
this.cache =
this.cacheManager.getEhcache(EXAM_CACHE_NAME);
}
Conversational State: View Exam Details
<view-state id=quot;examSelectedquot; view=quot;exam/detailsquot;>
<on-render>
<evaluate expression=quot;examService.findById(flowScope.examId)quot; result=quot;viewScope.examquot;/>
</on-render>
<transition on=quot;backquot; to=quot;chooseExamquot; />
<transition on=quot;returnquot; to=quot;returnquot; />
<transition on=quot;startExamquot; to=quot;startExamquot; />
</view-state>
public class CachingWrapperExamService implements ExamService {
/* as defined in ehcache.xml */
private static final String EXAM_CACHE_NAME = quot;examCachequot;;
private final CacheManager cacheManager;
private final Ehcache cache;
private final ExamService examService;
public CachingWrapperExamService(ExamService examService){
this.examService = examService;
this.cacheManager = CacheManager.getInstance();
this.cache = this.cacheManager.getEhcache(EXAM_CACHE_NAME);
}
// ...
public Exam findById(Long id) {
Exam exam = getCached(id);
if (exam == null){
exam = cache(this.examService.findById(id));
}
return exam;
}
// ...
}
Confidential – for information of designated recipient only. Copyright Terracotta 2006
40. Conversation State: Start Exam
<action-state id=quot;startExamquot;>
<on-entry>
<evaluate expression=quot;examSessionService.startExam(currentUser.name, requestParameters.examId)quot;/>
</on-entry>
<evaluate expression=quot;examSessionService.getFirstQuestion(currentUser.name).getId()quot;
result=quot;flowScope.questionIdquot;/>
<transition to=quot;showQuestionquot;/>
</action-state>
public class ExamSessionServiceImpl implements ExamSessionService {
private static final Logger logger = Logger.getLogger(ExamSessionServiceImpl.class);
// a map storing userName -> examSession mapping of currently active exams
@Root
private final Map<String, ExamSession> ongoingExams = new HashMap<String, ExamSession>();
private final ScheduledExecutorService examTimeoutExecutor = Executors.newScheduledThreadPool(1);
private final Map<String, Future> scheduledTimeOutTasks = new HashMap<String, Future>();
@Root
private final ReentrantReadWriteLock rwLock = new ReentrantReadWriteLock();
private final ExamService examService;
private final UserService userService;
@Autowired
public ExamSessionServiceImpl(final ExamService examService, final UserService userService) {
this.examService = examService;
this.userService = userService;
}
// ...
}
Confidential – for information of designated recipient only. Copyright Terracotta 2006
41. Conversation State: Start Exam (cont.)
public class ExamSessionServiceImpl implements ExamSessionService {
// ...
public ExamSession startExam(final String userName, final Long examId) throws ExamException {
final User user = userService.findByUserName(userName);
ExamSession prev = null;
try {
rwLock.writeLock().lock();
prev = ongoingExams.get(userName);
if (prev != null &&
prev.getExam().getId().longValue() != examId.longValue()) {
throw new ExamAlreadyInProgressException(userName, prev.getExam(), quot;Exam already in progress for user: quot;
+ userName);
}
// return previously ongoing exam if attempted to start same exam again
if (prev != null) return prev;
final Exam exam = examService.findById(examId);
final ExamSession session = new ExamSession();
session.setUser(user);
session.setExam(exam);
ongoingExams.put(userName, session);
// run the timeout after 2 secs from the actual time so that exam time out happens on the client first
// and give the user a chance to get ExamTimedOutException when exam times out on client
final Future timeoutTask = examTimeoutExecutor.schedule(new ExamTimeoutTask(this, userName), session
.getRemainingTimeInSeconds() + 2, TimeUnit.SECONDS);
scheduledTimeOutTasks.put(userName, timeoutTask);
return session;
} finally {
rwLock.writeLock().unlock();
}
}
// ...
}
Confidential – for information of designated recipient only. Copyright Terracotta 2006
42. Coherent View Of Global Data: View Ongoing Exams
@Controller
@RequestMapping(quot;/exam/ongoing.doquot;)
@RolesAllowed( { StandardAuthoritiesService.ADMINISTRATOR })
public class OngoingExamsController {
private final ExamSessionService service;
@Autowired
public OngoingExamsController(final ExamSessionService examSessionService) {
this.service = examSessionService;
}
@RequestMapping(method = RequestMethod.GET)
public ModelAndView listOngoingExams() {
final ModelAndView result = new ModelAndView(quot;exam/ongoingquot;);
result.addObject(quot;examSessionsquot;, service.getOngoingExams());
return result;
}
protected OngoingExamsController() {
// protected default constructor is needed for CGLib AOPx
service = null;
}
}
Confidential – for information of designated recipient only. Copyright Terracotta 2006
43. Coherent View Of Global Data: View Ongoing Exams
public class ExamSessionServiceImpl implements ExamSessionService {
private static final Logger logger = Logger.getLogger(ExamSessionServiceImpl.class);
// a map storing userName -> examSession mapping of currently active exams
@Root
private final Map<String, ExamSession> ongoingExams = new HashMap<String, ExamSession>();
private final ScheduledExecutorService examTimeoutExecutor = Executors.newScheduledThreadPool(1);
private final Map<String, Future> scheduledTimeOutTasks = new HashMap<String, Future>();
@Root
private final ReentrantReadWriteLock rwLock = new ReentrantReadWriteLock();
private final ExamService examService;
private final UserService userService;
// ...
public Collection<ExamSession> getOngoingExams() {
Collection<ExamSession> values = null;
try {
rwLock.readLock().lock();
values = new ArrayList(ongoingExams.values());
} finally {
rwLock.readLock().unlock();
}
return values;
}
// ...
}
Confidential – for information of designated recipient only. Copyright Terracotta 2006
44. Conversation State And Coherent View Of Global Data
Conversation State
– Memory-speed access to flow data
– Coherent cluster-wide
– Database-like durability for failover
– Handles failure of any/every node
– Completely transparent: no put-back on change
Coherent View Of Global Data
– Coherent cluster-wide
– No round-trip to database
» Ergo, no database abuse
» Ergo, no caching
» Ergo, no cache freshness probs
– Memory-speed read locks
– Simple data model
» ReentrantReadWriteLock for global thread access control
» Map holds global data
– Need to refine large data set access pattern
» @ 20K ongoing exams, single-page access is non-sensical
Confidential – for information of designated recipient only. Copyright Terracotta 2006
45. Case Study: Conversation Clustering
Confidential – for information of designated recipient only. Copyright Terracotta 2006
46. Large Publisher Gets Caught Down the Path with Oracle
Scaling Out or Up?
Confidential – for information of designated recipient only. Copyright Terracotta 2006
47. Breaking the Pattern without Leaving “Load-Balanced” World
• $1.5 Million DB & HW savings
• Doubled business
• More than halved database load
Confidential – for information of designated recipient only. Copyright Terracotta 2006
48. Results
Database was still the SoR which kept reporting and backup
simple
Scalability was increased by over 10X
Availability was not compromised since test data was still on disk,
but in memory-resident format instead of relational
…simple scalability + availability
Confidential – for information of designated recipient only. Copyright Terracotta 2006
49. When to Use Terracotta
Pattern: Window Onto Large Dataset
Isolate the JVMs from each other carefully
– ConcurrentHashMap of ConcurrentHashMaps
– One map per L1 JVM
If you must lock while updating, lock the value graph itself (fine-
grained) using ReadWriteLock
Enough Heap in each JVM to hold most data it needs
– High pre-fetch rate
You must come up with a way to load balance or partition or this
will thrash the network
Confidential – for information of designated recipient only. Copyright Terracotta 2006
50. When to Use Terracotta
Pattern: Data As A Service (impl coming December)
Service Oriented
Write a stateless get() router with ActiveMQ
– Send all messages to the get() router
– He hashes and sends the get() to appropriate stripe
– Transparent to service consumer
Fairly flat key / value pairs
– String or Integer key
– Value == shallow object graph (mostly primitives)
Key must hash evenly. Don’t hash spatially even though it is
tempting. (e.g., “New York customers” is worse than “A”)
Use MySQL to index keyspace and execute queries that return
vectors of key/value pairs to load
Update MySQL write-behind (another pattern)
Confidential – for information of designated recipient only. Copyright Terracotta 2006
51. When to Use Terracotta
Pattern: Cluster Membership
Want to know when L1 JVMs come and go
– Listen for Terracotta JMX events
– Catch L1 disconnect event and deterministically take over work (next worker
ID after failed worker)
Want to know when L2 JVMs fail over
– Listen for Terracotta JMX events
– If you don’t reconnect after certain time, system.exit() and use initd to restart
yourself
– Avoid disconnected mode
Always use a surviving L1 JVM to address a departing one. Do
not use System exit hooks! Clean up on restart if last JVM in the
cluster
Don’t requeue work, steal queue references instead
Confidential – for information of designated recipient only. Copyright Terracotta 2006
52. When to Use Terracotta
Pattern: Change Notification (impl coming soon)
Use this one for write-behind
Linked Blocking Queue in each L1 JVM
Share the LBQ via DSO in a CHM of queues
Have your changeable objects implement an iChangeable
interface. Call flush() method on change. You will have to
implement this method
Daemon thread takes batches of object references and calls flush
on each.
Idempotency is req’d to avoid transactions / JTA. Then peek
instead of take and take after flush().
Use Cluster membership (another pattern) to recover from L1 JVM
failure
Confidential – for information of designated recipient only. Copyright Terracotta 2006
53. When to Use Terracotta
Pattern: Shared Metadata
Composite Pattern…
Use EHCache + Linked Blocking Queue
Conversation Clustering pattern
Change Notification pattern
Confidential – for information of designated recipient only. Copyright Terracotta 2006
54. Summary
Simplicity, scalability, and availability can be friends
Write normal Java code that works across JVMs
Use clustered architecture patterns
– Conversation clustering/persistence
– Window onto large dataset
– Change notification
– Shared metadata
– Clustered membership and data management
Leave business data in the database, use durable NAM for
application state data
Centralized operational control: manage your application cluster
like you do your database
Terracotta is open source
– Free to use through production
– Enterprise subscription, training, and services available
Confidential – for information of designated recipient only. Copyright Terracotta 2006
55. Resources
Open Source (MPL-based) JVM-level clustering:
http://www.terracotta.org
Apress / Amazon.com: “Definitive Guide to Terracotta”
– By Alex Miller, Ari Zilka, Geert Bevin, Jonas Bonér, Orion Letizi, Taylor Gautier
Forums: http://forums.terracotta.org/
Enterprise Offerings: http://www.terracottatech.com/
Confidential – for information of designated recipient only. Copyright Terracotta 2006