1. MapR NFS
7/6/2012
© 2012 MapR Technologies Storage Service Architecture 1

2. MapR NFS
Agenda
• Before NFS
• Exercise: Using hadoop fs
• MapR NFS Overview
• NFS Mounting Models
• Exercise: Mount the Cluster to Itself and Copy a File
© 2012 MapR Technologies Storage Service Architecture 2

3. MapR NFS
Objectives
At the end of this module you will be able to:
• Describe the tools that are used with HDFS and their limitations
• List the features and benefits of MapR Direct Access NFS
• Identify several use cases for NFS on Hadoop
• Describe the different models for mounting NFS volumes with MapR
© 2012 MapR Technologies Storage Service Architecture 3

4. Before NFS
© 2012 MapR Technologies Storage Service Architecture 4

5. Before NFS
 Getting work done without MapR NFS
– Example: mining web logs for meaningful data
• First: get the logs into the cluster
– Do you stop writing to the logs?
– How long does it take?
– What do you do in the meantime?
hadoop fs -put, copyfromlocal
distcp
Flume
© 2012 MapR Technologies Storage Service Architecture 5

6. Before NFS
 Getting work done without MapR NFS
– Example: mining web logs for meaningful data
• Next: getting answers from the data
– What tools do you use to access data in your cluster?
– Who is using these tools?
hadoop fs
copytolocal
© 2012 MapR Technologies Storage Service Architecture 6

7. Exercise:
hadoop fs
© 2012 MapR Technologies Storage Service Architecture 7

8. MapR NFS Overview
© 2012 MapR Technologies Storage Service Architecture 8

9. Introducing Direct Access NFS
 Drag and drop
 Open files and apps
 Watch live data come in
 Fully read/write
 Multiple concurrent reads/writes
 Easy to get data in/out
 Same namespace paths as Hadoop
 Mount with
-o tcp,ver=3,nolock
© 2012 MapR Technologies Storage Service Architecture 9

10. Getting Data into the Cluster with NFS
 Simply mount your cluster filesystem locally
– /mapr/my.cluster.com/myvolume/…
 Redirect (or copy) logs to cluster filesystem
log data
© 2012 MapR Technologies Storage Service Architecture 10

11. Using NFS with Clients
 Mount your cluster filesystem locally
– /mapr/my.cluster.com/myvolume/…
 Access all of the data in your cluster with standard applications
– Double-click a file (spreadsheet, report, etc.)
• edit, save, etc.
– Run a perl script
– grep, sed, awk, shell scripts, etc.
 Use Hadoop apps (Hive, Pig, Hbase, etc.)
© 2012 MapR Technologies Storage Service Architecture 11

12. Using NFS with Clients
 Data is constantly changing
 How do I view the data at a point in time?
– Easy: take a snapshot
– cd to the snapshot volume
– View the data of interest
– Note: overhead for taking a snapshot: none to minimal
• The only write overhead is the data that changed
– Recall snapshot architecture
© 2012 MapR Technologies Storage Service Architecture 12

13. Using NFS with Legacy Applications
 What if I have legacy applications that work with my data?
– Log analysis, reporting tools, etc.
 Mount the cluster where your tools run
– Take a snapshot if necessary
– Then use your app just as before
© 2012 MapR Technologies Storage Service Architecture 13

14. NFS Mounting Models
© 2012 MapR Technologies Storage Service Architecture 14

15. NFS Mounting Models
 Export to the world
– NFS gateway runs on selected gateway hosts
 Local server
– NFS server runs on local host
– Enables local compression and check summing
 Export to self
– NFS gateway runs on all data nodes, mounted from localhost
© 2012 MapR Technologies Storage Service Architecture 15

16. Export to the World
NFS
NFS
Server
NFS
Server
NFS
Server
NFS Server
Client
© 2012 MapR Technologies Storage Service Architecture 16

17. Local Server
Application
NFS
Server
Client
Cluster
Nodes
© 2012 MapR Technologies Storage Service Architecture 17

18. Universal Export to Self
Cluster Nodes
Application
NFS
Cluster Server
Node
© 2012 MapR Technologies Storage Service Architecture 18

19. Nodes are Identical
Application
Application
NFS
NFS
Cluster Server
Node Cluster Server
Node
Application
NFS
Cluster Server
Node
© 2012 MapR Technologies Storage Service Architecture 19

20. Exercise:
Mount the Cluster to
Itself and Copy a File
© 2012 MapR Technologies Storage Service Architecture 20

21. Questions
© 2012 MapR Technologies Storage Service Architecture 21