Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
Hacking Infinispan: the new open source data grid meets NoSQL
1. Hacking Infinispan
Open Source Data Grids meets NoSQL
Manik Surtani
Founder and Project Lead, Infinispan
JBoss, by Red Hat Inc.
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
2. #infinispan
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
3. “Could data storage be the one thing that hampers true cloud
scalability and elasticity?”
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
4. Who is Manik?
• R&D Engineer, Red Hat Inc.
• Founder and project lead, Infinispan
• Project lead, JBoss Cache
• Frequent speaker on cloud computing and cloud data storage
• http://twitter.com/maniksurtani
• http://blog.infinispan.org
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
5. • What is Data-as-a-Service?
• Introducing Infinispan
• Implementing Data-as-a-Service with
Infinispan
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
6. Traditional 3-tier App
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
12. Virtualizing Data
• Some public services do exist
• Amazon RDS and SimpleDB
• FathomDB
• Cloundant
• MongoHQ
• etc.
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
13. What about private clouds?
• Not all cloud deployments
are public!
• Private cloud is very
important
• How can you build DaaS
yourself?
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
14. Characteristics of DaaS
• Elastic data
• Needs to scale with other tiers
• Response times should be linear
• Needs to be highly available
• Nodes will die! The service shouldn’t.
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
15. SQL and NoSQL DB
• Lack of distribution hampers
elasticity and HA
• These limitations can be worked
around
• ... but this isn’t trivial
• ... or cheap
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
16. SQL and NoSQL DB
• Lack of distribution hampers
elasticity and HA
• These limitations can be worked
around
• ... but this isn’t trivial
• ... or cheap
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
17. Distributed Data Grid
• Far better suited to elastic data
• Distributed by nature
• Highly available by nature
• A good building block for your data
service
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
18. API is king
• Apps should use their native data storage
APIs
• E.g., JPA (Java EE), ActiveRecord (Ruby), etc.
• Key/value stores too low level
• Akin to direct JDBC calls!
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
19. Introducing
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
20. • What is Infinispan?
• Open source (LGPL) in-memory Data Grid
• Written in Java and Scala
• Some concepts from Amazon Dynamo
• 2 usage modes
• Embedded
• Client-server
• memcached, REST and Hot Rod
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
23. • API
• Map-like key/value store
• Upcoming JPA-like layer
• Other high-level APIs being discussed in
the community e.g., ActiveRecord
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
24. • Consistent hash based distribution
• Self healing
• No single point of failure
• Highly concurrent
• MVCC locking
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
25. • Persistence
• Not just in memory!
• Write through and write behind
• Pluggable “drivers”
• Eviction and expiry
• Efficient, adaptive algorithms
• Addresses shortcomings of LRU & FIFO
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
26. • XA Transactions
• 2-phase commit based
• Deadlock detection algorithms
• Coming soon: Atomic Broadcast
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
27. • Map/Reduce
• In a pre-release state right now
• Please try out 5.0.0.ALPHA2 with these APIs!
• Querying
• Using Lucene and Hibernate Search to index
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
29. Client/Server Architecture
Supported Protocols
•REST
•Memcached
•Hot Rod
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
30. WTF is Hot Rod?
• Wire protocol for client server
communications
• Open
• Language independent
• Built-in failover and load balancing
• Smart routing
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
31. Server Endpoint Comparison
Client Smart Load Balancing/
Protocol Clustered?
Libraries Routing Failover
Any HTTP load
REST Text N/A Yes No
balancer
Only with predefined
Memcached Text Plenty Yes No
server list
Currently
Hot Rod Binary Yes Yes Dynamic
only Java
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
32. So is Infinispan a
data grid?
• In-memory
• P2P, distributed
• Low-latency, fast key/value
store
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
33. So is Infinispan a
data grid?
• In-memory
• P2P, distributed ... or is it a NoSQL
• Low-latency, fast key/value database?
store • Persistence
• Map/Reduce
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
34. So is Infinispan a
data grid?
• In-memory
• P2P, distributed ... or is it a NoSQL
• Low-latency, fast key/value database?
store • Persistence
• Map/Reduce ... or something
else?
• Querying support
• Transactional
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
35. So why is Infinispan sexy?
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
36. So why is Infinispan sexy?
• Transparent horizontal scalability
• Elastic in both directions
• Fast, low latency data access
• Ability to address a very large heap
• Cloud-ready datastore
• Not just for Java
• Open Source
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
37. DaaS with Infinispan
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
38. Architecture
Manage and Monitor
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
39. Where are we headed?
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
40. • 4.0.0 Starobrno
• Map-like API
• Consistent Hash based distribution
• Write-through, write-behind
• Eviction, expiration
• Management tooling
• REST API
• Hibernate 2nd Level Cache
• Released Feb 2010
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
41. • 4.1.0 Radegast
• Deadlock detection
• Client/Server protocols
• Memcached
• Hot Rod
• Smart clients using Hot Rod
• Lucene Directory implementation
• LIRS: adaptive, recency-based eviction policies
• Released August 2010
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
42. • 5.0.0 Pagoa
• JPA-like API
• Fine-grained replication
• Distributed code execution
• Map/reduce
• Virtual nodes for more even distribution
• In active development
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011
43. What have we learned?
• Elastic data is hard
• Public data services not always suitable
• Data grids make elastic storage easy
• Infinispan server endpoints help build elastic data tiers
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org
Saturday, 5 March 2011