Download Presentations http://ouo.io/XKLUj Student Guide & Best http://ouo.io/8u1RP

1. Spectrum Scale 4.1 System Administration
Quorums, Quorum Loss
and recovery
© Copyright IBM Corporation 2015

2. Unit objectives
After completing this unit, you should be able to:
• Review Quorums
• Review some Quorum best practices
• Describe recovery from quorum and cluster configuration node
failures
© Copyright IBM Corporation 2015

3. Spectrum Scale Quorum Nodes
Why do we need Quorum nodes?
• Quorum Nodes prevent multiple nodes from assuming the role
of the file system manager
• A majority of the quorum nodes MUST remain active in order
for the cluster to sustain normal file system activity.
• This prevents the cluster from becoming partitioned off where
two different nodes think they are each in control and corrupt
the file system
© Copyright IBM Corporation 2015

4. Best Practices: Quorum
• Quorum – set of 3-7 nodes use to determine the state of the
cluster
• Small subset of nodes designated as quorum nodes
• If there are only two nodes tiebreaker disks can be used to
determine quorum
• Recommendations
– Use an odd number of nodes
– Choose your most reliable nodes
– Choose nodes in separate: cabinets, power systems
– Use tiebreaker disks only on small clusters
– For small clusters, a simple VM can serve as a quorum node
– Quorum nodes should not be placed on extremely busy servers
© Copyright IBM Corporation 2015

5. RECOVERY SCENARIOS
© Copyright IBM Corporation 2015

6. How to recover: Node Quorum
How to recover from loss of majority for quorum nodes
• Scenario: You have 3 quorum nodes and 2 quorum nodes die.
– Nodes go into Arbitrating state
– Add 2 new quorum nodes
– mmchnode -–quorum –N node4,node5
– Now you have 5 quorum nodes and 3 are running
– Running nodes return to Active state
– Delete missing quorum nodes
– mmdelnode –N badnode1,badnode2
– or
– mmchnode –-nonquorum –N badnode1,badnode2
– Now you have 3 quorum nodes and 3 are running
– Running nodes remain in Active state
• You may have to clean up original quorum nodes once they
recover.
© Copyright IBM Corporation 2015

7. Recover from loss of primary configuration server
How to recover from loss of a primary configuration server
• Scenario: Primary cluster configuration server fails
– Secondary steps in
– Reassign primary
– mmchcluster –p newprimarynode
– To sync up the new primary
– mmchcluster –p LATEST
• Use the same process for failure of secondary cluster
configuration server.
© Copyright IBM Corporation 2015

8. How to recover: Loss of both configuration servers
How to recover from permanent loss of the primary configuration
and secondary servers
• Scenario: Primary and secondary cluster configuration servers
fail
– Cluster status
• Data is online
• mmgetstate -a no longer works
– Reassign both node configuration servers at the same time
– mmchcluster -p nodenewprimary –s nodenewsecondary
– Old servers still think they are in charge until
– mmchcluster –p Latest
© Copyright IBM Corporation 2015

9. I/O Concepts: Node config parameters (1 of 2)
• Spectrum Scale configuration parameters
• File system configuration
• Storage configuration
• Interconnect tuning
• Application design
© Copyright IBM Corporation 2015

10. I/O Concepts: Node config parameters (2 of 2)
• Spectrum Scale configuration parameters
• pagepool
• maxFilesToCache
• maxStatCache
• prefetchThreads
• socketRcvBufferSize
• socketSndBufferSize
• seqDiscardThreshhold
• Worker1threads
• generalWorkerThreads <<< Verify Role
• tokenMem <<< Verify Spelling
© Copyright IBM Corporation 2015

11. Performance tuning: File system parameters
• File system configuration
• -B Blocksize
• -n numnodes
• -l LogFileSize
• Storage Consideration
– RAID level
– RAID Strip Size (Segment Size)
– Storage Type: Flash, 10k RPM, 7,200 RPM
© Copyright IBM Corporation 2015

12. Performance tuning: Storage
• Storage configuration
– IO balance
• Even number of LUNS per path
• Even number of LUNS per NSD server
– RAID level
• Right level for IO pattern
– Cache configuration
• Read ahead not recommended for –j scatter
© Copyright IBM Corporation 2015

13. Performance tuning: Network configuration
• Interconnect selection and tuning
– TCP/IP tuning
– InfiniBand configuration
• Test Network performance with nsdperf
– In /usr/lpp/mmfs/samples/net/*
– README is documentation
– Some tools on Spectrum Scale Wiki
© Copyright IBM Corporation 2015

14. Performance tuning: Application design
• Make sure your applications are cluster aware
• Avoid and/or optimize for heavy operations
– Multiple nodes write locking a common entire file
• Use byte range locking
• Separate files
– Reduce use of IO heavy operations
• fsync()
– Some local node optimizations not applicable on cluster
• mmap
© Copyright IBM Corporation 2015

15. Direct I/O
• What is it?
– Bypass Spectrum Scale cache and transfer data directly from disk into
the user space buffer versus placing data in kernel memory.
• Why do I want it?
– Application with low file system cache hit rates
– Application with very large I/O
– When the application knows how to optimize IO
– Example: Oracle Database environment.
• How do I enable it?
– mmchattr –D yes <file_name>, or
– Specify O_DIRECT file access mode on the open() of the file.
© Copyright IBM Corporation 2015

16. RECOVERY SCENARIOS
© Copyright IBM Corporation 2015

17. How to recover: Node Quorum
How to recover from loss of majority for quorum nodes
• Scenario: You have 3 quorum nodes and 2 quorum nodes die.
– Nodes go into Arbitrating state
– Add 2 new quorum nodes
– mmchnode -–quorum –N node4,node5
– Now you have 5 quorum nodes and 3 are running
– Running nodes return to Active state
– Delete missing quorum nodes
– mmdelnode –N badnode1,badnode2
– or
– mmchnode –-nonquorum –N badnode1,badnode2
– Now you have 3 quorum nodes and 3 are running
– Running nodes remain in Active state
• You may have to clean up original quorum nodes once they
recover.
© Copyright IBM Corporation 2015

18. Recover from loss of primary configuration server
How to recover from loss of a primary configuration server
• Scenario: Primary cluster configuration server fails
– Secondary steps in
– Reassign primary
– mmchcluster –p newprimarynode
– To sync up the new primary
– mmchcluster –p LATEST
• Use the same process for failure of secondary cluster
configuration server.
© Copyright IBM Corporation 2015

19. How to recovery: Lose both configuration servers
How to recover from permanent loss of the primary configuration
and secondary servers
• Scenario: Primary and secondary cluster configuration servers
fail
– Cluster status
• Data is online
• mmgetstate -a no longer works
– Reassign both node configuration servers at the same time
– mmchcluster -p nodenewprimary –s nodenewsecondary
– Old servers still think they are in charge until
– mmchcluster –p Latest
© Copyright IBM Corporation 2015

20. Unit summary
Having completed this unit, you should be able to:
• Review some Spectrum Scale best practices
• Discuss approaches to Spectrum Scale performance tuning
• Describe recovery and recovery from quorum and cluster
configuration node failures
© Copyright IBM Corporation 2015