Hadoop Lens Toolkit is a product from Knowledge Lens Pvt Limited which is an end to end solution for hadoop cluster/data center migration. This is a very useful toolkit required for any customer who is using hadoop

1. HadoopLensToolkit
© Copyright 2014. All Rights Reserved

2. HadoopLensToolkitOverview
© Copyright 2014. All Rights Reserved
Cluster
Migrator
• Transfer data across firewalled cluster
• Migrate data across Hadoop distributions
• Compressed data transfer
• Distributed Parallel data transfer
• Re-start from the point of last successful transfer
Compressor
DC
Replicator
Folder
Comparator
• Identify changes to Hadoop Folders in Real Time
• Migrate the incremental changes across the Cluster
• Compressed data transfer for faster speed
• Detail Log for traceability of changes
• Compress and Backup data across firewalled cluster
• Configurable compression formats
• Target Format can be Parquet/AVRO/Text etc.
• High Speed Transfer
• Compares folders recursively across distributed
clusters
• Provides detailed report of comparison
• Provides comparative report for size, files, folders,
permissions, ownerships and replication factor
HadoopLens Toolkit
is the Industry’s
first high speed
secured data
transfer tool across
multiple clusters
High Speed Data Transfer

3. HadoopLens–Architecture
© Copyright 2014. All Rights Reserved
Source Hadoop Cluster Destination Hadoop Cluster
Edge Node1 Edge Node2
HLens Server
HLens Client
Resource
Manager
HLens Dispatch
Proxy ServerName Node
Data Node
Data Node
Data Node
Node Manager
Resource
Manager
Name Node
Data Node
Data Node
Data Node
Node Manager
rpc
rpc
http
rpc
http
Compressed high
speed parallel
data transfer
• Hadoop Cluster Migrator
• Hadoop Compressor
• Hadoop DC Replicator
• Hadoop Folder Comparator
• Hadoop SFTP Uploader

4. HadoopLens–FeatureList
© Copyright 2014. All Rights Reserved
• Transfer data across secured firewalled cluster. No access to cluster node required from remote cluster1
2 Transfer data across Hadoop Distributions. Supports Cloudera, Hortonworks, MapR, Pivotal and Asterdata clusters
3 Target file can be compressed to various standard compression formats like LZO,Bzip2, gzip etc.
4
5 Capability to merge multiple files into single Parquet file during transfer across the cluster
6 High Speed SFTP utility for parallel data ingestion directly to HDFS without any intermediate landing zone
7 Small file merge utility to ingesting small files into Hadoop by merging into multiple parquet files
8 Folder comparison report for comparing remote folders across Hadoop cluster
9 Complete Re-start capability from the point of last successful transfer
10 Real Time data replication with remote cluster
Target file can be of Parquet/AVRO/Text format

5. © Copyright 2014 . All Rights Reserved
ThankYou
Contact:
sudheesh@knowledgelens.com
sales@knowledgelens.com
Phone : +91 9916025302
Office : + 91 80 41102600
US Phone : +001 415 513 5991
Website : http://www.knowledgelens.com