Storing billions of images in a hybrid relational and NoSQL database using Oracle Active Data Guard and Oracle NoSQL

1. Storing billions of images
in a hybrid relational and NoSQL database
using Oracle Active Data Guard and Oracle NoSQL
Aris Prassinos

2. Who I am
• Oracle ACE 2007, Oracle ACE Director 2009
• Oracle Magazine Content Management Architect of the Year (2005)
• http://www.ArisPrassinos.com
Oracle Open World 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 2
© Aris Prassinos 2013

3. Agenda
• Introduce a real world use case
• Quick recap of relevant Oracle NoSQL concepts
• Present a solution that straddles relational and Oracle NoSQL databases
• Discuss things to consider and lessons learned
Oracle Open World 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 3
© Aris Prassinos 2013

4. The use case
• Application storing large amounts of binary data
• images, documents, short videos
• High value data
• data consistency and minimal data loss
• Write Once Read Many / occasional updates, deletes
• Random reads throughout database regardless of age of record
Oracle Open World 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 4
© Aris Prassinos 2013

5. Assumptions / Constraints
• High transaction volume with low latency
• Reads dominate writes
• High Availability using Active Standby datacenter
• Low latency between Primary and Standby datacenters
• Clients can arbitrarily connect to any datacenter
• System can tolerate a temporary loss of availability to guarantee consistency
Oracle Open World 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 5
© Aris Prassinos 2013

6. Traditional RDBMS approach
• Oracle Active Data Guard / Synchronous replication to Standby(s)
• Reader farm of Standby's to scale Reads
• Real-time intelligent routing of calls between Primary and Standby(s)
based on transaction requirements (e.g. read own writes) and Standby lag
http://www.oracle.com/technetwork/database/features/availability/morphotrak-132603.pdf
• Oracle Global Data Services to load balance Reads on Standbys
• Oracle 12c feature
• Minimal loss of availability during failover
• Third Observer site needed when two datacenters are used
Oracle Open World 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 6
© Aris Prassinos 2013

7. Why a hybrid solution
• High licensing cost for traditional RDBMS solution
• Other lower cost RDBMS not well suited for large scale image handling
• Concerns with storing images as regular files on the filesystem:
• durability / recovery
• consistency / integrity
• performance / scalability
• high availability / replication
• security
• Store the binary data and minimal metadata in Oracle NoSQL
and everything else in Oracle database
Oracle Open World 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 7
© Aris Prassinos 2013

8. Consistency between RDBMS and Oracle NoSQL
• Lack of transactions in Oracle NoSQL
• possibility of orphaned records left in Oracle NoSQL
• RDBMS is the source of truth
• Only query Oracle NoSQL using unique id obtained from RDBMS
• avoid queries on metadata in Oracle NoSQL
• Make NoSQL inserts idempotent (upserts)
• Oracle NoSQL may be ahead of an Active Data Guard Standby
• create a new version for each NoSQL update and store version# in RDBMS
• don't immediately delete from NoSQL but flag as deleted instead
Oracle Open World 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 8
© Aris Prassinos 2013

9. Oracle NoSQL key-value store
• Major key
: id, version
• Minor keys (columns): binary_data, binary_data.lob,
mime_type, size, signature, delete_flag
• Internal representation
• id/version/binary_data
=> data
• id/version/binary_data.lob => data
• id/version/signature
=> data
etc
• Note that Oracle NoSQL offers native streaming of large LOBs
but different columns (binary_data, binary_data.lob) are needed
to store both non-streamed and streamed LOBs under the same major key
• ACID semantics only for data within same Major key
Oracle Open World 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 9
© Aris Prassinos 2013

10. Oracle NoSQL Scalability
• Reader farm approach to scale Reads
• single Master / multiple Replicas architecture
• run-time load balancing of Reads based on real-time latency stats
• Oracle NoSQL latency aware driver
• Sharding to scale NoSQL Writes
• Multiple Oracle NoSQL Replication Groups
• Each shard (Replication Group) has its own Master and Replicas
• Transparent to the client
• A physical node may host the Master of one shard and the Replicas of other shards
• replicas may later be moved to dedicated physical nodes for scaling out
Oracle Open World 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 10
© Aris Prassinos 2013

11. Oracle NoSQL Master election
• Master is elected using PAXOS algorithm
• Master election held when a group of nodes forming a majority
does not have a Master
• when less than half of the nodes remain alive
the number that constitutes a majority can be manually overridden
using je.rep.electableGroupSizeOverride
• possible to end up with two Masters if former Master is in group
that consists of a minority of nodes after a network disconnect
• multiple Master problem solved by always requiring quorum writes
Oracle Open World 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 11
© Aris Prassinos 2013

12. Two datacenter configuration
• Nodes split equally between the two sites
• A third lightweight arbiter site needed to allow automatic master election
otherwise Standby site nodes cannot form a majority when Primary site lost
• Arbiter is an upcoming feature
• Arbiter placed at same site as Data Guard Observer
• otherwise manually set je.rep.electableGroupSizeOverride for manual failover
• By default NoSQL Master may not be on the same site as Data Guard Primary
• overridden using je.rep.node.priority for the preferred Master site nodes
Oracle Open World 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 12
© Aris Prassinos 2013

13. NoSQL Writes
• 1. Attempt a quorum write
(Durability.ReplicaAckPolicy.SIMPLE_MAJORITY)
In terms of CAP favor Latency over Consistency.
Possible inconsistencies can be handled when reading
2. If quorum write fails retry a few times to allow for potential new master election
3. If quorum write still fails decision depends on number of nodes (RF)
In terms of CAP favor Consistency over Availability.
If RF=2 attempt to write only to Master (Durability.ReplicaAckPolicy.NONE).
If RF>2 an administrator needs to intervene
since there is a possibility of having two masters if network partitioned
and writing without quorum could cause divergent transactions.
Oracle Open World 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 13
© Aris Prassinos 2013

14. NoSQL Reads
• 1. Read from any replica using unique id and version.
Due to quorum writes some replicas may not yet have the requested data
2. If data not found then attempt to read from Master.
We expect the data to exist since we query based on information
in the RDBMS which is the source of truth.
3. If Master not reachable return error.
• This approach achieves strong consistency, minimal latency, high availability
• most Reads to proceed when Master is temporarily unavailable
• only Reads resulting in no data found return error
Oracle Open World 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 14
© Aris Prassinos 2013

15. NoSQL log structured storage and backups
• Data always appended to log files up to a configurable size
• inserts, updates, deletes go to the latest log file
• Durability.SyncPolicy controls how often data written / flushed to disk
• old log files are immutable and may contain deleted or overwritten data
• Compaction process may move data from old log files to new ones
• when user defined percentage of valid records remains in a log file
• this is one of the primary candidates for tuning in a production system
• Snapshots and RSYNC can be used to perform incremental NoSQL backups
Oracle Open World 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 15
© Aris Prassinos 2013

16. NoSQL backups using replication
• NoSQL data since time of last snapshot lost after a restore
when only a single node used i.e. ReplicationFactor = 1
• Imperative to use NoSQL replication for loss-less recovery
• Possible to setup a node as a dedicated backup server for one or more nodes
• define each backup Rep Node as non electable master (i.e. always be Replica)
plan change-parameters ... configProperties="je.rep.node.priority=0"
Oracle Open World 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 16
© Aris Prassinos 2013

17. Conclusion
• Hybrid approach superior to just keeping pointers to binary data
stored on the filesystem
• replication, sharding, adjustable consistency / durability
• Strong data consistency achieved with careful application coding
• keeping in mind the consistency / availability / latency tradeoffs
• Very good architectural fit with Active Data Guard
• master / replica architecture, observer / arbiter third site
• Better suited for a replicated NoSQL configuration rather than a single node
• minimum of two server nodes and if necessary a third arbiter
Oracle Open World 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 17
© Aris Prassinos 2013