SlideShare a Scribd company logo
1 of 88
Download to read offline
Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

1 
Agenda

!
!
!
!
!

 About me
 Tips 1: Installation
 Tips 2: NFS Workloads
 Tips 3: Logging
 Conclusion

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

2 
About
Me
Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

3 
About me

!  Work at
!  Programmer
! 

as a ...

(Most recently)

! Also Interested in

!  Software Engineer
! Cloud Computing
! Big Data
! Something new technologies
! Supporting GlusterFS/Red Hat Storage Introduction
with Red Hat K.K.
Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

4 
About me

! GlusterFS since 2007 (v1.3.7)
!  for my internet crawler at first.

!  Love Gluster because of the ...
!  Potential
!  Performance
!  Code
!  Community

!  Introduced or introducing it into ...
!  Printer and scanner solution (field trial)
!  Email services
!  File storage services (WebDAV, NFS)
!  Backup services
!  Shared storage platform

!  A board member of the Gluster
Community
Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

5 
Getting started today!

See http://www.gluster.org/community/documentation/index.php/QuickStart
Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

6 
Tips 1
Installation
Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

7 
Tips 1: Installation
!   Question 1: Have you installed GlusterFS?

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

8 
Tips 1: Installation
!   Question 1: Have you installed GlusterFS?
!   Question 2: Did you find it easy?

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

9 
Tips 1: Installation
!   Question 1: Have you installed GlusterFS?
!   Question 2: Did you find it easy?
!   Question 3: Did you do it on each single node?

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

10 
Tips 1: Installation
!
!
!
!

 
 
 
 

Question 1: Have you installed GlusterFS?
Question 2: Did you find it easy?
Question 3: Did you do it on each single node?
Question 4: Don't you think you should take easier way?

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

11 
Tips 1: Installation
!
!
!
!
!

 
 
 
 
 

Question 1: Have you installed GlusterFS?
Question 2: Did you find it easy?
Question 3: Did you do it on each single node?
Question 4: Don't you think you should take easier way?
Question 5: Do you use deployment automation tools?

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

12 
Tips 1: Installation

Development team says that hand-to-hand construction of systems is inefficient so they want to
show an installation way with a deployment tool in the documents for installation.
Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

13 
Tips 1: Installation

Operational team says that they never want to learn the usage of deployment tools and never
trust them so they don't accept such a document and they will install the system by hand work.
Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

14 
Tips 1: Installation

My offer is that what development team should do is only copy the commands that the
deployment tool write on your stdout and paste them into the documents for operational team.
https://twitter.com/keithseahus/status/398453753352691714

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

15 
Tips 1: Installation

Deployment
Automation
Tools
Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

16 
Tips 1: Installation
A node for automated deployment
deployment
automation
tool

peer 1

peer 2

sshd

sshd

shell

3

4

N

shell
...

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

17 
Tips 1: Installation
A node for automated deployment
deployment
automation
tool

peer 1

peer 2

sshd

sshd

shell

3

4

N

shell
...

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

18 
Tips 1: Installation
A node for automated deployment
deployment
automation
tool

peer 1

peer 2

sshd

sshd

shell

3

4

N

shell
...

glusterfs

glusterfs

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

19 
Tips 1: Installation
A node for automated deployment
deployment
automation
tool

peer 1

peer 2

sshd

sshd

shell

3

4

N

shell
...

glusterfs

glusterfs

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

20 
Tips 1: Installation
A node for automated deployment
deployment
automation
tool

peer 1

peer 2

sshd

sshd

shell

3

4

N

shell
...

glusterfs

glusterfs

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

21 
Tips 1: Installation

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

22 
Tips 1: Installation

I use this mainly in my projects
written in Ruby so am used to it.

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

23 
Capistrano

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

24 
Capistrano
$ gem install capistrano
$ cap install
mkdir -p config/deploy
create config/deploy.rb
create config/deploy/staging.rb
create config/deploy/production.rb
mkdir -p lib/capistrano/tasks
Capified
$ tree .
.
├── Capfile
├── config
│   ├── deploy
│   │   ├── production.rb
│   │   └── staging.rb
│   └── deploy.rb
└── lib
└── capistrano
└── tasks

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

25 
capistrano-glusterfs

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

26 
capistrano-glusterfs
$ tree capistrano-glusterfs
capistrano-glusterfs
├── Capfile
├── Gemfile
├── Gemfile.lock
├── LICENSE
├── README.md
├── config
│   ├── deploy
│   │   ├── production.rb
│   │   ├── production.rb.example
│   │   ├── staging.rb
│   │   └── staging.rb.example
│   ├── deploy.rb
│   └── deploy.rb.example
└── lib
└── capistrano
└── tasks
├── gluster.cap
├── glusterd.cap
└── glusterfs.cap

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

27 
capistrano-glusterfs
$ tree capistrano-glusterfs
capistrano-glusterfs
├── Capfile
├── Gemfile
├── Gemfile.lock
├── LICENSE
├── README.md
├── config
│   ├── deploy
│   │   ├── production.rb ... settings for production environment
│   │   ├── production.rb.example
│   │   ├── staging.rb ... settings for staging environment
│   │   └── staging.rb.example
│   ├── deploy.rb ... global settings and main sequences
│   └── deploy.rb.example
└── lib
└── capistrano
└── tasks
├── gluster.cap ... tasks to execute gluster command remotely
├── glusterd.cap ... tasks to start/stop all remote glusterd
└── glusterfs.cap ... tasks for deploying GlusterFS

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

28 
capistrano-glusterfs
$ cat capistrano-glusterfs/config/deploy/staging.rb
set :stage, :staging
set :branch, "master"
server
server
server
server
server
server
server
server
server
server

'192.168.200.100',
'192.168.200.101',
'192.168.200.102',
'192.168.200.103',
'192.168.200.104',
'192.168.200.105',
'192.168.200.106',
'192.168.200.107',
'192.168.200.108',
'192.168.200.109',

roles:
roles:
roles:
roles:
roles:
roles:
roles:
roles:
roles:
roles:

%w(glusterfs
%w(glusterfs
%w(glusterfs
%w(glusterfs
%w(glusterfs
%w(glusterfs
%w(glusterfs
%w(glusterfs
%w(glusterfs
%w(glusterfs

glusterd gluster), user: 'deploy'
glusterd), user: 'deploy'
glusterd), user: 'deploy'
glusterd), user: 'deploy'
glusterd), user: 'deploy'
glusterd), user: 'deploy'
glusterd), user: 'deploy'
glusterd), user: 'deploy'
glusterd), user: 'deploy'
glusterd), user: 'deploy'

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

29 
capistrano-glusterfs
$ bundle exec cap -vT | grep -i "glusterfs:"
cap glusterfs:build
# Starting installation
cap glusterfs:compile
# Compile GlusterFS
cap glusterfs:configuration
# GlusterFS related configurations
cap glusterfs:dependancies
# Install GlusterFS dependancies
cap glusterfs:make_install
# Install GlusterFS
cap glusterfs:preparation
# Prepare installing GlusterFS
$ bundle exec cap -vT | grep -i "glusterd:"
cap glusterd:restart
# Restart glusterd
cap glusterd:start
# Start glusterd
cap glusterd:status
# Get the status of glusterd
cap glusterd:stop
# Stop glusterd
$ bundle exec cap -vT | grep -i "gluster:"
cap gluster:peer:help
# Help command for peer
cap gluster:peer:status
# list status of peers
cap gluster:pool:list
# list all the nodes in the pool
cap gluster:volume:help
# display help for the volume command
cap gluster:volume:info
# list information of all volumes
cap gluster:volume:list
# list all volumes in cluster
cap gluster:volume:status
# display status of all volume(s)/brick

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

30 
capistrano-glusterfs

http://www.capistranorb.com/documentation/getting-started/flow/

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

31 
capistrano-glusterfs

prepare and install
dependancies
before starting

build GlusterFS from
source and install it
after published
http://www.capistranorb.com/documentation/getting-started/flow/

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

32 
capistrano-glusterfs
task :preparation do
....
execute "sudo -i yum install -y wget git yum-priorities"
....
end

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

33 
capistrano-glusterfs
task :dependancies do
....
dependancies = %w[
openssh-clients
make
rpm-build
bison
flex
automake
libtool
ncurses-devel
readline-devel
openssl-devel
libxml2-devel
libibverbs-devel
libacl-devel
libattr-devel
python-devel
python-setuptools
lvm2-devel
systemtap-sdt-devel
libaio-devel
xfsprogs
]
Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

34 
capistrano-glusterfs
task :compile do
on roles(:glusterfs), in: :sequence, wait: 5 do
execute "cd #{DIR}; sh ./autogen.sh"
execute "cd #{DIR}; sh ./configure 
--prefix=#{INSTALL_PATH} 
--enable-bd-xlator 
--enable-fusermount 
--enable-systemtap 
--enable-debug"
execute "cd #{DIR}; make"
end
end

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

35 
capistrano-glusterfs
task :make_install do
on roles(:glusterfs), in: :sequence, wait: 5 do
execute "cd #{DIR}; sudo make install"
symlink = "/usr/local/glusterfs"
execute "sudo -i rm -f #{symlink}" if test("[ -L #{symlink} ]")
execute "sudo -i ln -snf #{INSTALL_PATH} #{symlink}"
end
end

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

36 
capistrano-glusterfs
$ git clone git@github.com:keithseahus/capistrano-glusterfs.git
$ cd capistrano-glusterfs
$ bundle exec cap staging deploy

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

37 
capistrano-glusterfs
$ git clone git@github.com:keithseahus/capistrano-glusterfs.git
$ cd capistrano-glusterfs
$ bundle exec cap staging deploy
DEBUG [e5bdd4a7] Running /usr/bin/env /bin/egrep '/bin' ~/.bashrc > /dev/null;
exit $? on 192.168.200.101
DEBUG [e5bdd4a7] Command: /bin/egrep '/bin' ~/.bashrc > /dev/null; exit $?
DEBUG [e5bdd4a7] Finished in 0.096 seconds with exit status 0 (successful).
INFO [a54cf65d] Running /usr/bin/env sudo -i yum install -y wget git yumpriorities on 192.168.200.101
DEBUG [a54cf65d] Command: sudo -i yum install -y wget git yum-priorities
DEBUG [a54cf65d] Loaded plugins: fastestmirror, priorities
DEBUG [a54cf65d] Loading mirror speeds from cached hostfile
epel/metalink
| 6.4 kB
00:00
DEBUG [a54cf65d]
* base: ftp.tsukuba.wide.ad.jp
DEBUG [a54cf65d]
* epel: ftp.kddilabs.jp
DEBUG [a54cf65d]
* extras: ftp.tsukuba.wide.ad.jp
DEBUG [a54cf65d]
* remi: rpms.famillecollet.com
DEBUG [a54cf65d]
* remi-php55: rpms.famillecollet.com
DEBUG [a54cf65d]
* remi-test: rpms.famillecollet.com
DEBUG [a54cf65d]
* rpmforge: ftp.kddilabs.jp
DEBUG [a54cf65d]
* updates: mirrors.tuna.tsinghua.edu.cn
...

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

38 
capistrano-glusterfs
...
DEBUG
DEBUG
DEBUG
DEBUG
DEBUG
DEBUG
DEBUG
DEBUG
DEBUG
DEBUG
DEBUG
DEBUG
DEBUG
DEBUG
DEBUG
DEBUG
DEBUG
...
INFO

[80062be9]
[80062be9]
[80062be9]
[80062be9]
[80062be9]
[80062be9]
[80062be9]
[80062be9]
[80062be9]
[80062be9]
[80062be9]
[80062be9]
[80062be9]
[80062be9]
[80062be9]
[80062be9]
[80062be9]

GlusterFS configure summary
===========================
FUSE client
: yes
Infiniband verbs
: no
epoll IO multiplex
: yes
argp-standalone
: no
fusermount
: yes
readline
: yes
georeplication
: yes
Linux-AIO
: yes
Enable Debug
: yes
systemtap
: yes
Block Device backend : yes
glupy
: yes
Use syslog
: yes
XML output
: yes
QEMU Block formats
: no

[a9e8de7e] Finished in 0.015 seconds with exit status 0 (successful).

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

39 
capistrano-glusterfs
$ bundle exec cap staging gluster:pool:list
INFO [201946fe] Running /usr/bin/env sudo -i gluster pool list on
192.168.200.101
DEBUG [201946fe] Command: sudo -i gluster pool list
DEBUG [201946fe] UUID
Hostname State
DEBUG [201946fe] d185fa19-d1a1-489c-a653-4510122f5278 localhostConnected
INFO [201946fe] Finished in 0.294 seconds with exit status 0 (successful).
$ bundle exec cap staging glusterd:status
INFO [cb482772] Running /usr/bin/env sudo -i /sbin/service glusterd status on
192.168.200.101
DEBUG [cb482772] Command: sudo -i /sbin/service glusterd status
INFO [cb482772] Finished in 0.151 seconds with exit status 0 (successful).
DEBUG [cb482772] glusterd (pid 25855) is running...
INFO [cb482772] Finished in 0.151 seconds with exit status 0 (successful).

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

40 
capistrano-glusterfs
$ bundle exec cap staging gluster:volume:help
INFO [4628fd38] Running /usr/bin/env sudo -i gluster volume help on
192.168.200.101
DEBUG [4628fd38] Command: sudo -i gluster volume help
DEBUG [4628fd38] volume info [all|<VOLNAME>] - list information of all volumes
DEBUG [4628fd38] volume create <NEW-VOLNAME> [stripe <COUNT>] [replica <COUNT>]
[device vg] [transport <tcp|rdma|tcp,rdma>] <NEW-BRICK> ... [force] - create a
new volume of specified type with mentioned bricks
DEBUG [4628fd38] volume delete <VOLNAME> - delete volume specified by <VOLNAME>
DEBUG [4628fd38] volume start <VOLNAME> [force] - start volume specified by
<VOLNAME>
DEBUG [4628fd38] volume stop <VOLNAME> [force] - stop volume specified by
<VOLNAME>
DEBUG [4628fd38] volume add-brick <VOLNAME> [<stripe|replica> <COUNT>] <NEWBRICK> ... [force] - add brick to volume <VOLNAME>
DEBUG [4628fd38] volume remove-brick <VOLNAME> [replica <COUNT>] <BRICK> ...
{start|stop|status|commit|force} - remove brick from volume <VOLNAME>
DEBUG [4628fd38] volume rebalance <VOLNAME> [fix-layout] {start|stop|status}
[force] - rebalance operations
DEBUG [4628fd38] volume replace-brick <VOLNAME> <BRICK> <NEW-BRICK> {start
[force]|pause|abort|status|commit [force]} - replace-brick operations
DEBUG [4628fd38] volume set <VOLNAME> <KEY> <VALUE> - set options for volume
<VOLNAME>
...
Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

41 
capistrano-glusterfs

capistrano-glusterfs is available here:
https://github.com/keithseahus/capistrano-glusterfs

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

42 
Tips 2
NFS Workloads

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

43 
Tips 2: NFS Workloads
!   Question 1: Do you mount GlusterFS in NFS?

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

44 
Tips 2: NFS Workloads
!   Question 1: Do you mount GlusterFS in NFS?
!   Question 2: Which is your main workload, write or read?

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

45 
Tips 2: NFS Workloads

peer 1
glusterd

glustershd

peer 2
glusterfs

d

shd

3

4

N

fs

...
glusterfsd
brick

brick

brick

fsd
brick

b

b

b

b

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

46 
Tips 2: NFS Workloads

TCP connection
in gluster protocol
peer 1
glusterd

glustershd

peer 2
glusterfs

d

shd

3

4

N

fs

...
glusterfsd
brick

brick

brick

fsd
brick

b

b

b

b

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

47 
Tips 2: NFS Workloads
NFS Client
user app
/mnt/nfs
vfs

page
cache
kernel

peer 1
glusterd

glustershd

peer 2
glusterfs

d

shd

3

4

N

fs

...
glusterfsd
brick

brick

brick

fsd
brick

b

b

b

b

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

48 
Tips 2: NFS Workloads
NFS Client
user app
/mnt/nfs
vfs

page
cache

TCP connection
in NFS v3

kernel

peer 1
glusterd

glustershd

peer 2
glusterfs

d

shd

3

4

N

fs

...
glusterfsd
brick

brick

brick

fsd
brick

b

b

b

b

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

49 
Tips 2: NFS Workloads
NFS Client
read access
to a file

user app
/mnt/nfs
vfs

page
cache
kernel

peer 1
glusterd

glustershd

peer 2
glusterfs

d

shd

3

4

N

fs

...
glusterfsd
brick

brick

brick

fsd
brick

b

b

b

b

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

50 
Tips 2: NFS Workloads
NFS Client
user app
/mnt/nfs
vfs

page
cache
kernel

performance/io-cache

peer 1
glusterd

glustershd

peer 2
glusterfs

d

shd

3

4

N

fs

...
glusterfsd
brick

brick

brick

fsd
brick

b

b

b

b

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

51 
Tips 2: NFS Workloads
NFS Client
user app
/mnt/nfs
vfs

activate cachefilesd
/var/cache/fscache

page
cache

cachefilesd

kernel

peer 1
glusterd

glustershd

peer 2
glusterfs

d

shd

3

4

N

fs

...
glusterfsd
brick

brick

brick

fsd
brick

b

b

b

b

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

52 
Tips 2: NFS Workloads
NFS Client
user app
/mnt/nfs
vfs

read files from its own RAM!

/var/cache/fscache

page
cache

cachefilesd

kernel

peer 1
glusterd

glustershd

peer 2
glusterfs

d

shd

3

4

N

fs

...
glusterfsd
brick

brick

brick

fsd
brick

b

b

b

b

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

53 
Tips 2: NFS Workloads
# NFS_SERVER=192.168.200.101
# MOUNT_POINT=/mnt/nfs

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

54 
Tips 2: NFS Workloads
# NFS_SERVER=192.168.200.101
# MOUNT_POINT=/mnt/nfs
# yum install -y cachefilesd

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

55 
Tips 2: NFS Workloads
# NFS_SERVER=192.168.200.101
# MOUNT_POINT=/mnt/nfs
# yum install -y cachefilesd
# if test `grep 'SELINUX=disabled' /etc/selinux/config > /dev/null; echo $?` -ne
0; then
sed -i 's/^secctx/#secctx/g' /etc/cachefilesd.conf
fi

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

56 
Tips 2: NFS Workloads
# NFS_SERVER=192.168.200.101
# MOUNT_POINT=/mnt/nfs
# yum install -y cachefilesd
# if test `grep 'SELINUX=disabled' /etc/selinux/config > /dev/null; echo $?` -ne
0; then
sed -i 's/^secctx/#secctx/g' /etc/cachefilesd.conf
fi
# chkconfig cachefilesd on
# service cachefilesd start

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

57 
Tips 2: NFS Workloads
# NFS_SERVER=192.168.200.101
# MOUNT_POINT=/mnt/nfs
# yum install -y cachefilesd
# if test `grep 'SELINUX=disabled' /etc/selinux/config > /dev/null; echo $?` -ne
0; then
sed -i 's/^secctx/#secctx/g' /etc/cachefilesd.conf
fi
# chkconfig cachefilesd on
# service cachefilesd start
# mount -t nfs -o fsc,vers=3 $NFS_SERVER $MOUNT_POINT

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

58 
Tips 2: NFS Workloads
# free -m
root@dev-centos:/root
total
Mem:
996
-/+ buffers/cache:
Swap:
2015

used
926
314
0

free
69
682
2015

shared
0

buffers
32

cached
579

# ls /var/cache/fscache/cache/@4a/I03nfs/@7f/Jc0000000000000cG0I00/@8d/
J1100000000000QFi24tY1EH900000000000M000000g00000100Kb000wFe000jt000oG3000000040
000g000000000/
@01 @0d @13 @1b @22 @35 @40 @50 @58 @61 @6e @78 @87 @a7 @b3 @b9
@d1 @d7 @e6
@06 @10 @17 @1e @25 @3c @4d @52 @5d @63 @6f @81 @97 @a9 @b6 @c1
@d3 @dc @f1
@0a @11 @1a @21 @2b @3d @4f @57 @5f @66 @73 @82 @9a @ae @b7 @cc
@d4 @de @fe
# du -h /var/cache/fscache/cache/@4a/I03nfs/@7f/Jc0000000000000cG0I00/@8d/
J1100000000000QFi24tY1EH900000000000M000000g00000100Kb000wFe000jt000oG3000000040
000g000000000/
186M /var/cache/fscache/cache/@4a/I03nfs/@7f/Jc0000000000000cG0I00/@8d/
J1100000000000QFi24tY1EH900000000000M000000g00000100Kb000wFe000jt000oG3000000040
000g000000000/
Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

59 
Tips 2: NFS Workloads
nodes:
VirtualBox
CentOS 6.4 x86_64
4 vCPUs, Core i7 2.4GHz
2GB RAM
Striped SSDs

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

60 
Tips 2: NFS Workloads
nodes:
VirtualBox
CentOS 6.4 x86_64
4 vCPUs, Core i7 2.4GHz
2GB RAM
Striped SSDs
postmark 1.51
set size 1048576 10485760
set number 500
set transactions 1000
set location /mnt/nfs/rep2
set subdirectories 100
set read 8192
set write 8192
set buffering true

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

61 
Tips 2: NFS Workloads
nodes:
VirtualBox
CentOS 6.4 x86_64
4 vCPUs, Core i7 2.4GHz
2GB RAM
Striped SSDs
MB/sec (read)
postmark 1.51
set size 1048576 10485760
set number 500
set transactions 1000
set location /mnt/nfs/rep2
set subdirectories 100
set read 8192
set write 8192
set buffering true

↓Oops...
Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

62 
Tips 2: NFS Workloads

Okay... but this is still a hypothesis.
We need your feedback!

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

63 
Tips 3
Logging

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

64 
Tips 3: Logging
!   Question 1: Do you monitor GlusterFS logs?

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

65 
Tips 3: Logging
!   Question 1: Do you monitor GlusterFS logs?
!   Question 2: Do you utilize the logs?

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

66 
Tips 3: Logging
!   Question 1: Do you monitor GlusterFS logs?
!   Question 2: Do you utilize the logs?
!   Question 3: Do you use a log collector or aggregator?

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

67 
Tips 3: Logging

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

68 
Tips 3: Logging

I pick it up because:
•  easy to install
•  easy to configure
•  make unstructured logs structured
•  collect logs from multiple resources
•  output aggregated and structured
logs into a target as you prefer
•  easy to create a module
•  high performance
•  well maintained
Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

69 
Tips 3: Logging
NFS Client
user app
/mnt/nfs
vfs

/var/cache/fscache

page
cache

cachefilesd

kernel

peer 1
glusterd

glustershd

peer 2
glusterfs

d

shd

3

4

N

fs

...
glusterfsd
brick

brick

brick

fsd
brick

b

b

b

b

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

70 
Tips 3: Logging
NFS Client

Log server

user app
/mnt/nfs
vfs

/var/cache/fscache

page
cache

cachefilesd

kernel

peer 1
glusterd

glustershd

peer 2
glusterfs

d

shd

3

4

N

fs

...
glusterfsd
brick

brick

brick

fsd
brick

b

b

b

b

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

71 
Tips 3: Logging
NFS Client

Log server

user app
/mnt/nfs
vfs

/var/cache/fscache

page
cache

cachefilesd

kernel

peer 1
glusterd

glustershd

peer 2
glusterfs

d

shd

3

4

N

fs

...
glusterfsd
brick

brick

brick

fsd
brick

b

b

b

b

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

72 
Tips 3: Logging
NFS Client

Log server

user app
/mnt/nfs
vfs

/var/cache/fscache

page
cache

cachefilesd

kernel

peer 1
glusterd

glustershd

peer 2
glusterfs

d

shd

3

4

N

fs

...
glusterfsd
brick

brick

brick

fsd
brick

b

b

b

b

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

73 
Tips 3: Logging
NFS Client

Log server

user app
/mnt/nfs
vfs

/var/cache/fscache

page
cache

cachefilesd

kernel

peer 1
glusterd

glustershd

peer 2
glusterfs

d

shd

3

4

N

fs

...
glusterfsd
brick

brick

brick

fsd
brick

b

b

b

b

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

74 
Tips 3: Logging
NFS Client

Log server

user app
/mnt/nfs
vfs

/var/cache/fscache

page
cache

cachefilesd

kernel

peer 1
glusterd

glustershd

peer 2
glusterfs

d

shd

3

4

N

fs

...
glusterfsd
brick

brick

brick

fsd
brick

b

b

b

b

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

75 
Tips 3: Logging
/etc/td-agent/td-agent.conf (on the log server)
<source>
type forward
port 24224
bind 0.0.0.0
</source>
<match filtered.glusterd.**>
type file
path /home/keith/var/log/fluent/glusterd
</match>

Today I will get the td-agent
output the logs into a file.

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

76 
Tips 3: Logging
/etc/td-agent/td-agent.conf (on the GlusterFS peers)
<source>
type tail
path /var/log/glusterfs/usr-local-glusterfs-3.4.0beta2-etc-glusterfs-glusterd.vol.log
pos_file /var/log/td-agent/usr-local-glusterfs-3.4.0beta2-etc-glusterfs-glusterd.vol.log.pos
tag glusterd
format /^(?:[(?<date_ymd>[0-9]{4}-[01][0-9]-[0-3][0-9]) (?<time_hms>[0-2][0-9]:[0-5][0-9]:[0-6]
[0-9]).(?<time_usec>[0-9]{6})]) (?<log_level>[TDINWECA]) (?:[(?<source_file_name>[^[]:/ ]*):(?
<source_line>[0-9]*):(?<function_name>[^[]:/ ]*)]) (?<component>[^[]:/ ]*): *(?<message>.*)$/
rotate_wait 5
</source>
<match glusterd.**>
type hostname
key_name host
add_prefix filtered
</match>

Too complex to input
and maintain!

<match filtered.glusterd.**>
type forward
send_timeout 60s
recover_wait 10s
heartbeat_interval 1s
phi_threshold 8
hard_timeout 60s
<server>
…

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

77 
Tips 3: Logging

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

78 
Tips 3: Logging

You can specify log levels
to forward.

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

79 
Tips 3: Logging

You can specify each field
name to output in JSON.

Comment out if you don't
need the fields.

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

80 
Tips 3: Logging

Parse error should be
treated as a failure of
fluent-plugin-glusterfs.

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

81 
Tips 3: Logging
It makes the logs structured in JSON.
[root@glusterfs-unstable-01 plugin]# tail -3 /var/log/glusterfs/usr-local-glusterfs-3.4.0beta2-etcglusterfs-glusterd.vol.log
[2013-07-07 15:52:43.778961] I [socket.c:3495:socket_init] 0-management: using system polling thread
[2013-07-07 15:52:43.779397] W [socket.c:514:__socket_rwv] 0-management: readv failed (No data
available)
[2013-07-07 15:52:43.779668] I [socket.c:2236:socket_event_handler] 0-transport: disconnecting now

[root@dev-centos ~]# grep glusterfs-unstable-01 /home/keith/var/log/fluent/glusterd.
20130708.b4e0ed6e32655d35f | tail -3
2013-07-08T00:52:43+09:00
filtered.glusterd
{"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"778961","log_level":"I","source_file_name":
"socket.c","source_line":"3495","function_name":"socket_init","component":"0management","message":"using system polling thread","host":"glusterfs-unstable-01.localdomain"}
2013-07-08T00:52:43+09:00
filtered.glusterd
{"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"779397","log_level":"W","source_file_name":
"socket.c","source_line":"514","function_name":"__socket_rwv","component":"0management","message":"readv failed (No data available)","host":"glusterfs-unstable-01.localdomain"}
2013-07-08T00:52:43+09:00
filtered.glusterd
{"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"779668","log_level":"I","source_file_name":
"socket.c","source_line":"2236","function_name":"socket_event_handler","component":"0transport","message":"disconnecting now","host":"glusterfs-unstable-01.localdomain"}
Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

82 
Tips 3: Logging
The logs on each peer could be output in a single log file.
[root@dev-centos ~]# grep glusterfs-unstable-01 /home/keith/var/log/fluent/glusterd.
20130708.b4e0ed6e32655d35f | tail -3
2013-07-08T00:52:43+09:00
filtered.glusterd
{"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"778961","log_level":"I","source_file_name":
"socket.c","source_line":"3495","function_name":"socket_init","component":"0management","message":"using system polling thread","host":"glusterfs-unstable-01.localdomain"}
2013-07-08T00:52:43+09:00
filtered.glusterd
{"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"779397","log_level":"W","source_file_name":
"socket.c","source_line":"514","function_name":"__socket_rwv","component":"0management","message":"readv failed (No data available)","host":"glusterfs-unstable-01.localdomain"}
2013-07-08T00:52:43+09:00
filtered.glusterd
{"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"779668","log_level":"I","source_file_name":
"socket.c","source_line":"2236","function_name":"socket_event_handler","component":"0transport","message":"disconnecting now","host":"glusterfs-unstable-01.localdomain"}
[root@dev-centos ~]# grep glusterfs-unstable-02 /home/keith/var/log/fluent/glusterd.
20130708.b4e0ed6e32655d35f | tail -3
2013-07-08T00:52:43+09:00
filtered.glusterd
{"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"759502","log_level":"I","source_file_name":
"socket.c","source_line":"3495","function_name":"socket_init","component":"0management","message":"using system polling thread","host":"glusterfs-unstable-02.localdomain"}
2013-07-08T00:52:43+09:00
filtered.glusterd
{"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"759838","log_level":"W","source_file_name":
"socket.c","source_line":"514","function_name":"__socket_rwv","component":"0management","message":"readv failed (No data available)","host":"glusterfs-unstable-02.localdomain"}
2013-07-08T00:52:43+09:00
filtered.glusterd
{"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"760057","log_level":"I","source_file_name":
"socket.c","source_line":"2236","function_name":"socket_event_handler","component":"0transport","message":"disconnecting now","host":"glusterfs-unstable-02.localdomain"}
Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

83 
Tips 3: Logging

Also available on Gluster Forge:
https://forge.gluster.org/fluent-plugin-glusterfs

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

84 
Tips 3: Logging

More detailed explanation in Japanese is here!:
http://www.nttpc.co.jp/company/r_and_d/technology/gluster_fs_fluentd.html
Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

85 
Conclusion

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

86 
Conclusion

!  Deployment automation tools are
useful to install and operate
GlusterFS on multiple nodes.
!  With cachefilesd, NFS can might
be more efficient under heavy
read workload.
!  Aggregating GlusterFS logs is easy
enough and you can utilize them.

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

87 
To contact us, e-mail here -> storage-contact@nttpc.co.jp
Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   88 

More Related Content

What's hot

Velocity 2017 Performance analysis superpowers with Linux eBPF
Velocity 2017 Performance analysis superpowers with Linux eBPFVelocity 2017 Performance analysis superpowers with Linux eBPF
Velocity 2017 Performance analysis superpowers with Linux eBPF
Brendan Gregg
 
LizardFS-WhitePaper-Eng-v3.9.2-web
LizardFS-WhitePaper-Eng-v3.9.2-webLizardFS-WhitePaper-Eng-v3.9.2-web
LizardFS-WhitePaper-Eng-v3.9.2-web
Szymon Haly
 
Kvm performance optimization for ubuntu
Kvm performance optimization for ubuntuKvm performance optimization for ubuntu
Kvm performance optimization for ubuntu
Sim Janghoon
 

What's hot (20)

Lagopus presentation on 14th Annual ON*VECTOR International Photonics Workshop
Lagopus presentation on 14th Annual ON*VECTOR International Photonics WorkshopLagopus presentation on 14th Annual ON*VECTOR International Photonics Workshop
Lagopus presentation on 14th Annual ON*VECTOR International Photonics Workshop
 
Comparison of foss distributed storage
Comparison of foss distributed storageComparison of foss distributed storage
Comparison of foss distributed storage
 
Velocity 2017 Performance analysis superpowers with Linux eBPF
Velocity 2017 Performance analysis superpowers with Linux eBPFVelocity 2017 Performance analysis superpowers with Linux eBPF
Velocity 2017 Performance analysis superpowers with Linux eBPF
 
LizardFS-WhitePaper-Eng-v3.9.2-web
LizardFS-WhitePaper-Eng-v3.9.2-webLizardFS-WhitePaper-Eng-v3.9.2-web
LizardFS-WhitePaper-Eng-v3.9.2-web
 
Introduction to eBPF and XDP
Introduction to eBPF and XDPIntroduction to eBPF and XDP
Introduction to eBPF and XDP
 
NUSE (Network Stack in Userspace) at #osio
NUSE (Network Stack in Userspace) at #osioNUSE (Network Stack in Userspace) at #osio
NUSE (Network Stack in Userspace) at #osio
 
Introduction to DPDK
Introduction to DPDKIntroduction to DPDK
Introduction to DPDK
 
L3HA-VRRP-20141201
L3HA-VRRP-20141201L3HA-VRRP-20141201
L3HA-VRRP-20141201
 
Kvm performance optimization for ubuntu
Kvm performance optimization for ubuntuKvm performance optimization for ubuntu
Kvm performance optimization for ubuntu
 
LISA18: Hidden Linux Metrics with Prometheus eBPF Exporter
LISA18: Hidden Linux Metrics with Prometheus eBPF ExporterLISA18: Hidden Linux Metrics with Prometheus eBPF Exporter
LISA18: Hidden Linux Metrics with Prometheus eBPF Exporter
 
Kernel Recipes 2019 - XDP closer integration with network stack
Kernel Recipes 2019 -  XDP closer integration with network stackKernel Recipes 2019 -  XDP closer integration with network stack
Kernel Recipes 2019 - XDP closer integration with network stack
 
Comparison of-foss-distributed-storage
Comparison of-foss-distributed-storageComparison of-foss-distributed-storage
Comparison of-foss-distributed-storage
 
Improvements in GlusterFS for Virtualization usecase
Improvements in GlusterFS for Virtualization usecaseImprovements in GlusterFS for Virtualization usecase
Improvements in GlusterFS for Virtualization usecase
 
The 7 Deadly Sins of Packet Processing - Venky Venkatesan and Bruce Richardson
The 7 Deadly Sins of Packet Processing - Venky Venkatesan and Bruce RichardsonThe 7 Deadly Sins of Packet Processing - Venky Venkatesan and Bruce Richardson
The 7 Deadly Sins of Packet Processing - Venky Venkatesan and Bruce Richardson
 
Performance Tuning EC2 Instances
Performance Tuning EC2 InstancesPerformance Tuning EC2 Instances
Performance Tuning EC2 Instances
 
OpenStack networking juno l3 h-a, dvr
OpenStack networking   juno l3 h-a, dvrOpenStack networking   juno l3 h-a, dvr
OpenStack networking juno l3 h-a, dvr
 
Performance comparison of Distributed File Systems on 1Gbit networks
Performance comparison of Distributed File Systems on 1Gbit networksPerformance comparison of Distributed File Systems on 1Gbit networks
Performance comparison of Distributed File Systems on 1Gbit networks
 
Dpdk applications
Dpdk applicationsDpdk applications
Dpdk applications
 
The n00bs guide to ovs dpdk
The n00bs guide to ovs dpdkThe n00bs guide to ovs dpdk
The n00bs guide to ovs dpdk
 
Ceph Day Beijing - Ceph RDMA Update
Ceph Day Beijing - Ceph RDMA UpdateCeph Day Beijing - Ceph RDMA Update
Ceph Day Beijing - Ceph RDMA Update
 

Similar to Gluster Cloud Night in Tokyo 2013 -- Tips for getting started

Final ProjectFinal Project Details Description Given a spec.docx
Final ProjectFinal Project Details Description  Given a spec.docxFinal ProjectFinal Project Details Description  Given a spec.docx
Final ProjectFinal Project Details Description Given a spec.docx
AKHIL969626
 

Similar to Gluster Cloud Night in Tokyo 2013 -- Tips for getting started (20)

Gluster in Japan 2012-2013
Gluster in Japan 2012-2013Gluster in Japan 2012-2013
Gluster in Japan 2012-2013
 
Universal Userland
Universal UserlandUniversal Userland
Universal Userland
 
Containers with systemd-nspawn
Containers with systemd-nspawnContainers with systemd-nspawn
Containers with systemd-nspawn
 
snortinstallguide
snortinstallguidesnortinstallguide
snortinstallguide
 
Final ProjectFinal Project Details Description Given a spec.docx
Final ProjectFinal Project Details Description  Given a spec.docxFinal ProjectFinal Project Details Description  Given a spec.docx
Final ProjectFinal Project Details Description Given a spec.docx
 
DevOpSec_DockerNPodMan-20230220.pdf
DevOpSec_DockerNPodMan-20230220.pdfDevOpSec_DockerNPodMan-20230220.pdf
DevOpSec_DockerNPodMan-20230220.pdf
 
The internet of $h1t
The internet of $h1tThe internet of $h1t
The internet of $h1t
 
Backtrack Manual Part4
Backtrack Manual Part4Backtrack Manual Part4
Backtrack Manual Part4
 
bettercap.pdf
bettercap.pdfbettercap.pdf
bettercap.pdf
 
Linux Security and How Web Browser Sandboxes Really Work (NDC Oslo 2017)
Linux Security  and How Web Browser Sandboxes Really Work (NDC Oslo 2017)Linux Security  and How Web Browser Sandboxes Really Work (NDC Oslo 2017)
Linux Security and How Web Browser Sandboxes Really Work (NDC Oslo 2017)
 
Arun
ArunArun
Arun
 
Building a Gateway Server
Building a Gateway ServerBuilding a Gateway Server
Building a Gateway Server
 
Backtrack Manual Part8
Backtrack Manual Part8Backtrack Manual Part8
Backtrack Manual Part8
 
Interview questions
Interview questionsInterview questions
Interview questions
 
12 tricks to avoid hackers breaks your CI / CD
12 tricks to avoid hackers breaks your  CI / CD12 tricks to avoid hackers breaks your  CI / CD
12 tricks to avoid hackers breaks your CI / CD
 
From Mediasoup WebRTC to Livekit Self-Hosted .pdf
From Mediasoup WebRTC to  Livekit Self-Hosted .pdfFrom Mediasoup WebRTC to  Livekit Self-Hosted .pdf
From Mediasoup WebRTC to Livekit Self-Hosted .pdf
 
A tale of a rouge router firmware
A tale of a rouge router firmwareA tale of a rouge router firmware
A tale of a rouge router firmware
 
Attacking IoT Devices from a Web Perspective - Linux Day
Attacking IoT Devices from a Web Perspective - Linux Day Attacking IoT Devices from a Web Perspective - Linux Day
Attacking IoT Devices from a Web Perspective - Linux Day
 
Ubuntu Core 技术详解
Ubuntu Core 技术详解Ubuntu Core 技术详解
Ubuntu Core 技术详解
 
Tutorial to setup OpenStreetMap tileserver with customized boundaries of India
Tutorial to setup OpenStreetMap tileserver with customized boundaries of IndiaTutorial to setup OpenStreetMap tileserver with customized boundaries of India
Tutorial to setup OpenStreetMap tileserver with customized boundaries of India
 

More from Keisuke Takahashi

最新技術動向 GlusterFS (仮想化DAY, Internet Week 2011)
最新技術動向 GlusterFS (仮想化DAY, Internet Week 2011)最新技術動向 GlusterFS (仮想化DAY, Internet Week 2011)
最新技術動向 GlusterFS (仮想化DAY, Internet Week 2011)
Keisuke Takahashi
 

More from Keisuke Takahashi (13)

Azure Database for PostgreSQL 入門 (PostgreSQL Conference Japan 2021)
Azure Database for PostgreSQL 入門 (PostgreSQL Conference Japan 2021)Azure Database for PostgreSQL 入門 (PostgreSQL Conference Japan 2021)
Azure Database for PostgreSQL 入門 (PostgreSQL Conference Japan 2021)
 
パーフェクト"Elixir情報収集"
パーフェクト"Elixir情報収集"パーフェクト"Elixir情報収集"
パーフェクト"Elixir情報収集"
 
GlusterFS Masakari Talks
GlusterFS Masakari TalksGlusterFS Masakari Talks
GlusterFS Masakari Talks
 
GlusterFS Updates (and more) in 第六回クラウドストレージ研究会
GlusterFS Updates (and more) in 第六回クラウドストレージ研究会GlusterFS Updates (and more) in 第六回クラウドストレージ研究会
GlusterFS Updates (and more) in 第六回クラウドストレージ研究会
 
Big Data入門に見せかけたFluentd入門
Big Data入門に見せかけたFluentd入門Big Data入門に見せかけたFluentd入門
Big Data入門に見せかけたFluentd入門
 
Creating a shared storage service with GlusterFS
Creating a shared storage service with GlusterFSCreating a shared storage service with GlusterFS
Creating a shared storage service with GlusterFS
 
GlusterFSとInfiniBandの小話
GlusterFSとInfiniBandの小話GlusterFSとInfiniBandの小話
GlusterFSとInfiniBandの小話
 
GlusterFS 技術と動向 2of2
GlusterFS 技術と動向 2of2GlusterFS 技術と動向 2of2
GlusterFS 技術と動向 2of2
 
GlusterFS 技術と動向 1of2
GlusterFS 技術と動向 1of2GlusterFS 技術と動向 1of2
GlusterFS 技術と動向 1of2
 
最新技術動向 GlusterFS (仮想化DAY, Internet Week 2011)
最新技術動向 GlusterFS (仮想化DAY, Internet Week 2011)最新技術動向 GlusterFS (仮想化DAY, Internet Week 2011)
最新技術動向 GlusterFS (仮想化DAY, Internet Week 2011)
 
GlusterFS As an Object Storage
GlusterFS As an Object StorageGlusterFS As an Object Storage
GlusterFS As an Object Storage
 
GlusterFS モジュール超概論
GlusterFS モジュール超概論GlusterFS モジュール超概論
GlusterFS モジュール超概論
 
GlusterFS座談会テクニカルセッション
GlusterFS座談会テクニカルセッションGlusterFS座談会テクニカルセッション
GlusterFS座談会テクニカルセッション
 

Recently uploaded

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@
 

Recently uploaded (20)

Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CVReal Time Object Detection Using Open CV
Real Time Object Detection Using Open CV
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 

Gluster Cloud Night in Tokyo 2013 -- Tips for getting started

  • 1. Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   1 
  • 2. Agenda ! ! ! ! !  About me  Tips 1: Installation  Tips 2: NFS Workloads  Tips 3: Logging  Conclusion Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   2 
  • 3. About Me Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   3 
  • 4. About me !  Work at !  Programmer !  as a ... (Most recently) ! Also Interested in !  Software Engineer ! Cloud Computing ! Big Data ! Something new technologies ! Supporting GlusterFS/Red Hat Storage Introduction with Red Hat K.K. Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   4 
  • 5. About me ! GlusterFS since 2007 (v1.3.7) !  for my internet crawler at first. !  Love Gluster because of the ... !  Potential !  Performance !  Code !  Community !  Introduced or introducing it into ... !  Printer and scanner solution (field trial) !  Email services !  File storage services (WebDAV, NFS) !  Backup services !  Shared storage platform !  A board member of the Gluster Community Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   5 
  • 6. Getting started today! See http://www.gluster.org/community/documentation/index.php/QuickStart Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   6 
  • 7. Tips 1 Installation Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   7 
  • 8. Tips 1: Installation !   Question 1: Have you installed GlusterFS? Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   8 
  • 9. Tips 1: Installation !   Question 1: Have you installed GlusterFS? !   Question 2: Did you find it easy? Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   9 
  • 10. Tips 1: Installation !   Question 1: Have you installed GlusterFS? !   Question 2: Did you find it easy? !   Question 3: Did you do it on each single node? Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   10 
  • 11. Tips 1: Installation ! ! ! !         Question 1: Have you installed GlusterFS? Question 2: Did you find it easy? Question 3: Did you do it on each single node? Question 4: Don't you think you should take easier way? Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   11 
  • 12. Tips 1: Installation ! ! ! ! !           Question 1: Have you installed GlusterFS? Question 2: Did you find it easy? Question 3: Did you do it on each single node? Question 4: Don't you think you should take easier way? Question 5: Do you use deployment automation tools? Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   12 
  • 13. Tips 1: Installation Development team says that hand-to-hand construction of systems is inefficient so they want to show an installation way with a deployment tool in the documents for installation. Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   13 
  • 14. Tips 1: Installation Operational team says that they never want to learn the usage of deployment tools and never trust them so they don't accept such a document and they will install the system by hand work. Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   14 
  • 15. Tips 1: Installation My offer is that what development team should do is only copy the commands that the deployment tool write on your stdout and paste them into the documents for operational team. https://twitter.com/keithseahus/status/398453753352691714 Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   15 
  • 16. Tips 1: Installation Deployment Automation Tools Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   16 
  • 17. Tips 1: Installation A node for automated deployment deployment automation tool peer 1 peer 2 sshd sshd shell 3 4 N shell ... Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   17 
  • 18. Tips 1: Installation A node for automated deployment deployment automation tool peer 1 peer 2 sshd sshd shell 3 4 N shell ... Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   18 
  • 19. Tips 1: Installation A node for automated deployment deployment automation tool peer 1 peer 2 sshd sshd shell 3 4 N shell ... glusterfs glusterfs Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   19 
  • 20. Tips 1: Installation A node for automated deployment deployment automation tool peer 1 peer 2 sshd sshd shell 3 4 N shell ... glusterfs glusterfs Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   20 
  • 21. Tips 1: Installation A node for automated deployment deployment automation tool peer 1 peer 2 sshd sshd shell 3 4 N shell ... glusterfs glusterfs Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   21 
  • 22. Tips 1: Installation Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   22 
  • 23. Tips 1: Installation I use this mainly in my projects written in Ruby so am used to it. Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   23 
  • 24. Capistrano Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   24 
  • 25. Capistrano $ gem install capistrano $ cap install mkdir -p config/deploy create config/deploy.rb create config/deploy/staging.rb create config/deploy/production.rb mkdir -p lib/capistrano/tasks Capified $ tree . . ├── Capfile ├── config │   ├── deploy │   │   ├── production.rb │   │   └── staging.rb │   └── deploy.rb └── lib └── capistrano └── tasks Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   25 
  • 26. capistrano-glusterfs Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   26 
  • 27. capistrano-glusterfs $ tree capistrano-glusterfs capistrano-glusterfs ├── Capfile ├── Gemfile ├── Gemfile.lock ├── LICENSE ├── README.md ├── config │   ├── deploy │   │   ├── production.rb │   │   ├── production.rb.example │   │   ├── staging.rb │   │   └── staging.rb.example │   ├── deploy.rb │   └── deploy.rb.example └── lib └── capistrano └── tasks ├── gluster.cap ├── glusterd.cap └── glusterfs.cap Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   27 
  • 28. capistrano-glusterfs $ tree capistrano-glusterfs capistrano-glusterfs ├── Capfile ├── Gemfile ├── Gemfile.lock ├── LICENSE ├── README.md ├── config │   ├── deploy │   │   ├── production.rb ... settings for production environment │   │   ├── production.rb.example │   │   ├── staging.rb ... settings for staging environment │   │   └── staging.rb.example │   ├── deploy.rb ... global settings and main sequences │   └── deploy.rb.example └── lib └── capistrano └── tasks ├── gluster.cap ... tasks to execute gluster command remotely ├── glusterd.cap ... tasks to start/stop all remote glusterd └── glusterfs.cap ... tasks for deploying GlusterFS Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   28 
  • 29. capistrano-glusterfs $ cat capistrano-glusterfs/config/deploy/staging.rb set :stage, :staging set :branch, "master" server server server server server server server server server server '192.168.200.100', '192.168.200.101', '192.168.200.102', '192.168.200.103', '192.168.200.104', '192.168.200.105', '192.168.200.106', '192.168.200.107', '192.168.200.108', '192.168.200.109', roles: roles: roles: roles: roles: roles: roles: roles: roles: roles: %w(glusterfs %w(glusterfs %w(glusterfs %w(glusterfs %w(glusterfs %w(glusterfs %w(glusterfs %w(glusterfs %w(glusterfs %w(glusterfs glusterd gluster), user: 'deploy' glusterd), user: 'deploy' glusterd), user: 'deploy' glusterd), user: 'deploy' glusterd), user: 'deploy' glusterd), user: 'deploy' glusterd), user: 'deploy' glusterd), user: 'deploy' glusterd), user: 'deploy' glusterd), user: 'deploy' Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   29 
  • 30. capistrano-glusterfs $ bundle exec cap -vT | grep -i "glusterfs:" cap glusterfs:build # Starting installation cap glusterfs:compile # Compile GlusterFS cap glusterfs:configuration # GlusterFS related configurations cap glusterfs:dependancies # Install GlusterFS dependancies cap glusterfs:make_install # Install GlusterFS cap glusterfs:preparation # Prepare installing GlusterFS $ bundle exec cap -vT | grep -i "glusterd:" cap glusterd:restart # Restart glusterd cap glusterd:start # Start glusterd cap glusterd:status # Get the status of glusterd cap glusterd:stop # Stop glusterd $ bundle exec cap -vT | grep -i "gluster:" cap gluster:peer:help # Help command for peer cap gluster:peer:status # list status of peers cap gluster:pool:list # list all the nodes in the pool cap gluster:volume:help # display help for the volume command cap gluster:volume:info # list information of all volumes cap gluster:volume:list # list all volumes in cluster cap gluster:volume:status # display status of all volume(s)/brick Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   30 
  • 32. capistrano-glusterfs prepare and install dependancies before starting build GlusterFS from source and install it after published http://www.capistranorb.com/documentation/getting-started/flow/ Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   32 
  • 33. capistrano-glusterfs task :preparation do .... execute "sudo -i yum install -y wget git yum-priorities" .... end Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   33 
  • 34. capistrano-glusterfs task :dependancies do .... dependancies = %w[ openssh-clients make rpm-build bison flex automake libtool ncurses-devel readline-devel openssl-devel libxml2-devel libibverbs-devel libacl-devel libattr-devel python-devel python-setuptools lvm2-devel systemtap-sdt-devel libaio-devel xfsprogs ] Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   34 
  • 35. capistrano-glusterfs task :compile do on roles(:glusterfs), in: :sequence, wait: 5 do execute "cd #{DIR}; sh ./autogen.sh" execute "cd #{DIR}; sh ./configure --prefix=#{INSTALL_PATH} --enable-bd-xlator --enable-fusermount --enable-systemtap --enable-debug" execute "cd #{DIR}; make" end end Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   35 
  • 36. capistrano-glusterfs task :make_install do on roles(:glusterfs), in: :sequence, wait: 5 do execute "cd #{DIR}; sudo make install" symlink = "/usr/local/glusterfs" execute "sudo -i rm -f #{symlink}" if test("[ -L #{symlink} ]") execute "sudo -i ln -snf #{INSTALL_PATH} #{symlink}" end end Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   36 
  • 37. capistrano-glusterfs $ git clone git@github.com:keithseahus/capistrano-glusterfs.git $ cd capistrano-glusterfs $ bundle exec cap staging deploy Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   37 
  • 38. capistrano-glusterfs $ git clone git@github.com:keithseahus/capistrano-glusterfs.git $ cd capistrano-glusterfs $ bundle exec cap staging deploy DEBUG [e5bdd4a7] Running /usr/bin/env /bin/egrep '/bin' ~/.bashrc > /dev/null; exit $? on 192.168.200.101 DEBUG [e5bdd4a7] Command: /bin/egrep '/bin' ~/.bashrc > /dev/null; exit $? DEBUG [e5bdd4a7] Finished in 0.096 seconds with exit status 0 (successful). INFO [a54cf65d] Running /usr/bin/env sudo -i yum install -y wget git yumpriorities on 192.168.200.101 DEBUG [a54cf65d] Command: sudo -i yum install -y wget git yum-priorities DEBUG [a54cf65d] Loaded plugins: fastestmirror, priorities DEBUG [a54cf65d] Loading mirror speeds from cached hostfile epel/metalink | 6.4 kB 00:00 DEBUG [a54cf65d] * base: ftp.tsukuba.wide.ad.jp DEBUG [a54cf65d] * epel: ftp.kddilabs.jp DEBUG [a54cf65d] * extras: ftp.tsukuba.wide.ad.jp DEBUG [a54cf65d] * remi: rpms.famillecollet.com DEBUG [a54cf65d] * remi-php55: rpms.famillecollet.com DEBUG [a54cf65d] * remi-test: rpms.famillecollet.com DEBUG [a54cf65d] * rpmforge: ftp.kddilabs.jp DEBUG [a54cf65d] * updates: mirrors.tuna.tsinghua.edu.cn ... Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   38 
  • 39. capistrano-glusterfs ... DEBUG DEBUG DEBUG DEBUG DEBUG DEBUG DEBUG DEBUG DEBUG DEBUG DEBUG DEBUG DEBUG DEBUG DEBUG DEBUG DEBUG ... INFO [80062be9] [80062be9] [80062be9] [80062be9] [80062be9] [80062be9] [80062be9] [80062be9] [80062be9] [80062be9] [80062be9] [80062be9] [80062be9] [80062be9] [80062be9] [80062be9] [80062be9] GlusterFS configure summary =========================== FUSE client : yes Infiniband verbs : no epoll IO multiplex : yes argp-standalone : no fusermount : yes readline : yes georeplication : yes Linux-AIO : yes Enable Debug : yes systemtap : yes Block Device backend : yes glupy : yes Use syslog : yes XML output : yes QEMU Block formats : no [a9e8de7e] Finished in 0.015 seconds with exit status 0 (successful). Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   39 
  • 40. capistrano-glusterfs $ bundle exec cap staging gluster:pool:list INFO [201946fe] Running /usr/bin/env sudo -i gluster pool list on 192.168.200.101 DEBUG [201946fe] Command: sudo -i gluster pool list DEBUG [201946fe] UUID Hostname State DEBUG [201946fe] d185fa19-d1a1-489c-a653-4510122f5278 localhostConnected INFO [201946fe] Finished in 0.294 seconds with exit status 0 (successful). $ bundle exec cap staging glusterd:status INFO [cb482772] Running /usr/bin/env sudo -i /sbin/service glusterd status on 192.168.200.101 DEBUG [cb482772] Command: sudo -i /sbin/service glusterd status INFO [cb482772] Finished in 0.151 seconds with exit status 0 (successful). DEBUG [cb482772] glusterd (pid 25855) is running... INFO [cb482772] Finished in 0.151 seconds with exit status 0 (successful). Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   40 
  • 41. capistrano-glusterfs $ bundle exec cap staging gluster:volume:help INFO [4628fd38] Running /usr/bin/env sudo -i gluster volume help on 192.168.200.101 DEBUG [4628fd38] Command: sudo -i gluster volume help DEBUG [4628fd38] volume info [all|<VOLNAME>] - list information of all volumes DEBUG [4628fd38] volume create <NEW-VOLNAME> [stripe <COUNT>] [replica <COUNT>] [device vg] [transport <tcp|rdma|tcp,rdma>] <NEW-BRICK> ... [force] - create a new volume of specified type with mentioned bricks DEBUG [4628fd38] volume delete <VOLNAME> - delete volume specified by <VOLNAME> DEBUG [4628fd38] volume start <VOLNAME> [force] - start volume specified by <VOLNAME> DEBUG [4628fd38] volume stop <VOLNAME> [force] - stop volume specified by <VOLNAME> DEBUG [4628fd38] volume add-brick <VOLNAME> [<stripe|replica> <COUNT>] <NEWBRICK> ... [force] - add brick to volume <VOLNAME> DEBUG [4628fd38] volume remove-brick <VOLNAME> [replica <COUNT>] <BRICK> ... {start|stop|status|commit|force} - remove brick from volume <VOLNAME> DEBUG [4628fd38] volume rebalance <VOLNAME> [fix-layout] {start|stop|status} [force] - rebalance operations DEBUG [4628fd38] volume replace-brick <VOLNAME> <BRICK> <NEW-BRICK> {start [force]|pause|abort|status|commit [force]} - replace-brick operations DEBUG [4628fd38] volume set <VOLNAME> <KEY> <VALUE> - set options for volume <VOLNAME> ... Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   41 
  • 42. capistrano-glusterfs capistrano-glusterfs is available here: https://github.com/keithseahus/capistrano-glusterfs Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   42 
  • 43. Tips 2 NFS Workloads Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   43 
  • 44. Tips 2: NFS Workloads !   Question 1: Do you mount GlusterFS in NFS? Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   44 
  • 45. Tips 2: NFS Workloads !   Question 1: Do you mount GlusterFS in NFS? !   Question 2: Which is your main workload, write or read? Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   45 
  • 46. Tips 2: NFS Workloads peer 1 glusterd glustershd peer 2 glusterfs d shd 3 4 N fs ... glusterfsd brick brick brick fsd brick b b b b Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   46 
  • 47. Tips 2: NFS Workloads TCP connection in gluster protocol peer 1 glusterd glustershd peer 2 glusterfs d shd 3 4 N fs ... glusterfsd brick brick brick fsd brick b b b b Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   47 
  • 48. Tips 2: NFS Workloads NFS Client user app /mnt/nfs vfs page cache kernel peer 1 glusterd glustershd peer 2 glusterfs d shd 3 4 N fs ... glusterfsd brick brick brick fsd brick b b b b Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   48 
  • 49. Tips 2: NFS Workloads NFS Client user app /mnt/nfs vfs page cache TCP connection in NFS v3 kernel peer 1 glusterd glustershd peer 2 glusterfs d shd 3 4 N fs ... glusterfsd brick brick brick fsd brick b b b b Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   49 
  • 50. Tips 2: NFS Workloads NFS Client read access to a file user app /mnt/nfs vfs page cache kernel peer 1 glusterd glustershd peer 2 glusterfs d shd 3 4 N fs ... glusterfsd brick brick brick fsd brick b b b b Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   50 
  • 51. Tips 2: NFS Workloads NFS Client user app /mnt/nfs vfs page cache kernel performance/io-cache peer 1 glusterd glustershd peer 2 glusterfs d shd 3 4 N fs ... glusterfsd brick brick brick fsd brick b b b b Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   51 
  • 52. Tips 2: NFS Workloads NFS Client user app /mnt/nfs vfs activate cachefilesd /var/cache/fscache page cache cachefilesd kernel peer 1 glusterd glustershd peer 2 glusterfs d shd 3 4 N fs ... glusterfsd brick brick brick fsd brick b b b b Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   52 
  • 53. Tips 2: NFS Workloads NFS Client user app /mnt/nfs vfs read files from its own RAM! /var/cache/fscache page cache cachefilesd kernel peer 1 glusterd glustershd peer 2 glusterfs d shd 3 4 N fs ... glusterfsd brick brick brick fsd brick b b b b Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   53 
  • 54. Tips 2: NFS Workloads # NFS_SERVER=192.168.200.101 # MOUNT_POINT=/mnt/nfs Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   54 
  • 55. Tips 2: NFS Workloads # NFS_SERVER=192.168.200.101 # MOUNT_POINT=/mnt/nfs # yum install -y cachefilesd Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   55 
  • 56. Tips 2: NFS Workloads # NFS_SERVER=192.168.200.101 # MOUNT_POINT=/mnt/nfs # yum install -y cachefilesd # if test `grep 'SELINUX=disabled' /etc/selinux/config > /dev/null; echo $?` -ne 0; then sed -i 's/^secctx/#secctx/g' /etc/cachefilesd.conf fi Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   56 
  • 57. Tips 2: NFS Workloads # NFS_SERVER=192.168.200.101 # MOUNT_POINT=/mnt/nfs # yum install -y cachefilesd # if test `grep 'SELINUX=disabled' /etc/selinux/config > /dev/null; echo $?` -ne 0; then sed -i 's/^secctx/#secctx/g' /etc/cachefilesd.conf fi # chkconfig cachefilesd on # service cachefilesd start Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   57 
  • 58. Tips 2: NFS Workloads # NFS_SERVER=192.168.200.101 # MOUNT_POINT=/mnt/nfs # yum install -y cachefilesd # if test `grep 'SELINUX=disabled' /etc/selinux/config > /dev/null; echo $?` -ne 0; then sed -i 's/^secctx/#secctx/g' /etc/cachefilesd.conf fi # chkconfig cachefilesd on # service cachefilesd start # mount -t nfs -o fsc,vers=3 $NFS_SERVER $MOUNT_POINT Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   58 
  • 59. Tips 2: NFS Workloads # free -m root@dev-centos:/root total Mem: 996 -/+ buffers/cache: Swap: 2015 used 926 314 0 free 69 682 2015 shared 0 buffers 32 cached 579 # ls /var/cache/fscache/cache/@4a/I03nfs/@7f/Jc0000000000000cG0I00/@8d/ J1100000000000QFi24tY1EH900000000000M000000g00000100Kb000wFe000jt000oG3000000040 000g000000000/ @01 @0d @13 @1b @22 @35 @40 @50 @58 @61 @6e @78 @87 @a7 @b3 @b9 @d1 @d7 @e6 @06 @10 @17 @1e @25 @3c @4d @52 @5d @63 @6f @81 @97 @a9 @b6 @c1 @d3 @dc @f1 @0a @11 @1a @21 @2b @3d @4f @57 @5f @66 @73 @82 @9a @ae @b7 @cc @d4 @de @fe # du -h /var/cache/fscache/cache/@4a/I03nfs/@7f/Jc0000000000000cG0I00/@8d/ J1100000000000QFi24tY1EH900000000000M000000g00000100Kb000wFe000jt000oG3000000040 000g000000000/ 186M /var/cache/fscache/cache/@4a/I03nfs/@7f/Jc0000000000000cG0I00/@8d/ J1100000000000QFi24tY1EH900000000000M000000g00000100Kb000wFe000jt000oG3000000040 000g000000000/ Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   59 
  • 60. Tips 2: NFS Workloads nodes: VirtualBox CentOS 6.4 x86_64 4 vCPUs, Core i7 2.4GHz 2GB RAM Striped SSDs Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   60 
  • 61. Tips 2: NFS Workloads nodes: VirtualBox CentOS 6.4 x86_64 4 vCPUs, Core i7 2.4GHz 2GB RAM Striped SSDs postmark 1.51 set size 1048576 10485760 set number 500 set transactions 1000 set location /mnt/nfs/rep2 set subdirectories 100 set read 8192 set write 8192 set buffering true Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   61 
  • 62. Tips 2: NFS Workloads nodes: VirtualBox CentOS 6.4 x86_64 4 vCPUs, Core i7 2.4GHz 2GB RAM Striped SSDs MB/sec (read) postmark 1.51 set size 1048576 10485760 set number 500 set transactions 1000 set location /mnt/nfs/rep2 set subdirectories 100 set read 8192 set write 8192 set buffering true ↓Oops... Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   62 
  • 63. Tips 2: NFS Workloads Okay... but this is still a hypothesis. We need your feedback! Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   63 
  • 64. Tips 3 Logging Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   64 
  • 65. Tips 3: Logging !   Question 1: Do you monitor GlusterFS logs? Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   65 
  • 66. Tips 3: Logging !   Question 1: Do you monitor GlusterFS logs? !   Question 2: Do you utilize the logs? Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   66 
  • 67. Tips 3: Logging !   Question 1: Do you monitor GlusterFS logs? !   Question 2: Do you utilize the logs? !   Question 3: Do you use a log collector or aggregator? Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   67 
  • 68. Tips 3: Logging Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   68 
  • 69. Tips 3: Logging I pick it up because: •  easy to install •  easy to configure •  make unstructured logs structured •  collect logs from multiple resources •  output aggregated and structured logs into a target as you prefer •  easy to create a module •  high performance •  well maintained Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   69 
  • 70. Tips 3: Logging NFS Client user app /mnt/nfs vfs /var/cache/fscache page cache cachefilesd kernel peer 1 glusterd glustershd peer 2 glusterfs d shd 3 4 N fs ... glusterfsd brick brick brick fsd brick b b b b Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   70 
  • 71. Tips 3: Logging NFS Client Log server user app /mnt/nfs vfs /var/cache/fscache page cache cachefilesd kernel peer 1 glusterd glustershd peer 2 glusterfs d shd 3 4 N fs ... glusterfsd brick brick brick fsd brick b b b b Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   71 
  • 72. Tips 3: Logging NFS Client Log server user app /mnt/nfs vfs /var/cache/fscache page cache cachefilesd kernel peer 1 glusterd glustershd peer 2 glusterfs d shd 3 4 N fs ... glusterfsd brick brick brick fsd brick b b b b Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   72 
  • 73. Tips 3: Logging NFS Client Log server user app /mnt/nfs vfs /var/cache/fscache page cache cachefilesd kernel peer 1 glusterd glustershd peer 2 glusterfs d shd 3 4 N fs ... glusterfsd brick brick brick fsd brick b b b b Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   73 
  • 74. Tips 3: Logging NFS Client Log server user app /mnt/nfs vfs /var/cache/fscache page cache cachefilesd kernel peer 1 glusterd glustershd peer 2 glusterfs d shd 3 4 N fs ... glusterfsd brick brick brick fsd brick b b b b Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   74 
  • 75. Tips 3: Logging NFS Client Log server user app /mnt/nfs vfs /var/cache/fscache page cache cachefilesd kernel peer 1 glusterd glustershd peer 2 glusterfs d shd 3 4 N fs ... glusterfsd brick brick brick fsd brick b b b b Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   75 
  • 76. Tips 3: Logging /etc/td-agent/td-agent.conf (on the log server) <source> type forward port 24224 bind 0.0.0.0 </source> <match filtered.glusterd.**> type file path /home/keith/var/log/fluent/glusterd </match> Today I will get the td-agent output the logs into a file. Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   76 
  • 77. Tips 3: Logging /etc/td-agent/td-agent.conf (on the GlusterFS peers) <source> type tail path /var/log/glusterfs/usr-local-glusterfs-3.4.0beta2-etc-glusterfs-glusterd.vol.log pos_file /var/log/td-agent/usr-local-glusterfs-3.4.0beta2-etc-glusterfs-glusterd.vol.log.pos tag glusterd format /^(?:[(?<date_ymd>[0-9]{4}-[01][0-9]-[0-3][0-9]) (?<time_hms>[0-2][0-9]:[0-5][0-9]:[0-6] [0-9]).(?<time_usec>[0-9]{6})]) (?<log_level>[TDINWECA]) (?:[(?<source_file_name>[^[]:/ ]*):(? <source_line>[0-9]*):(?<function_name>[^[]:/ ]*)]) (?<component>[^[]:/ ]*): *(?<message>.*)$/ rotate_wait 5 </source> <match glusterd.**> type hostname key_name host add_prefix filtered </match> Too complex to input and maintain! <match filtered.glusterd.**> type forward send_timeout 60s recover_wait 10s heartbeat_interval 1s phi_threshold 8 hard_timeout 60s <server> … Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   77 
  • 78. Tips 3: Logging Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   78 
  • 79. Tips 3: Logging You can specify log levels to forward. Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   79 
  • 80. Tips 3: Logging You can specify each field name to output in JSON. Comment out if you don't need the fields. Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   80 
  • 81. Tips 3: Logging Parse error should be treated as a failure of fluent-plugin-glusterfs. Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   81 
  • 82. Tips 3: Logging It makes the logs structured in JSON. [root@glusterfs-unstable-01 plugin]# tail -3 /var/log/glusterfs/usr-local-glusterfs-3.4.0beta2-etcglusterfs-glusterd.vol.log [2013-07-07 15:52:43.778961] I [socket.c:3495:socket_init] 0-management: using system polling thread [2013-07-07 15:52:43.779397] W [socket.c:514:__socket_rwv] 0-management: readv failed (No data available) [2013-07-07 15:52:43.779668] I [socket.c:2236:socket_event_handler] 0-transport: disconnecting now [root@dev-centos ~]# grep glusterfs-unstable-01 /home/keith/var/log/fluent/glusterd. 20130708.b4e0ed6e32655d35f | tail -3 2013-07-08T00:52:43+09:00 filtered.glusterd {"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"778961","log_level":"I","source_file_name": "socket.c","source_line":"3495","function_name":"socket_init","component":"0management","message":"using system polling thread","host":"glusterfs-unstable-01.localdomain"} 2013-07-08T00:52:43+09:00 filtered.glusterd {"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"779397","log_level":"W","source_file_name": "socket.c","source_line":"514","function_name":"__socket_rwv","component":"0management","message":"readv failed (No data available)","host":"glusterfs-unstable-01.localdomain"} 2013-07-08T00:52:43+09:00 filtered.glusterd {"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"779668","log_level":"I","source_file_name": "socket.c","source_line":"2236","function_name":"socket_event_handler","component":"0transport","message":"disconnecting now","host":"glusterfs-unstable-01.localdomain"} Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   82 
  • 83. Tips 3: Logging The logs on each peer could be output in a single log file. [root@dev-centos ~]# grep glusterfs-unstable-01 /home/keith/var/log/fluent/glusterd. 20130708.b4e0ed6e32655d35f | tail -3 2013-07-08T00:52:43+09:00 filtered.glusterd {"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"778961","log_level":"I","source_file_name": "socket.c","source_line":"3495","function_name":"socket_init","component":"0management","message":"using system polling thread","host":"glusterfs-unstable-01.localdomain"} 2013-07-08T00:52:43+09:00 filtered.glusterd {"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"779397","log_level":"W","source_file_name": "socket.c","source_line":"514","function_name":"__socket_rwv","component":"0management","message":"readv failed (No data available)","host":"glusterfs-unstable-01.localdomain"} 2013-07-08T00:52:43+09:00 filtered.glusterd {"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"779668","log_level":"I","source_file_name": "socket.c","source_line":"2236","function_name":"socket_event_handler","component":"0transport","message":"disconnecting now","host":"glusterfs-unstable-01.localdomain"} [root@dev-centos ~]# grep glusterfs-unstable-02 /home/keith/var/log/fluent/glusterd. 20130708.b4e0ed6e32655d35f | tail -3 2013-07-08T00:52:43+09:00 filtered.glusterd {"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"759502","log_level":"I","source_file_name": "socket.c","source_line":"3495","function_name":"socket_init","component":"0management","message":"using system polling thread","host":"glusterfs-unstable-02.localdomain"} 2013-07-08T00:52:43+09:00 filtered.glusterd {"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"759838","log_level":"W","source_file_name": "socket.c","source_line":"514","function_name":"__socket_rwv","component":"0management","message":"readv failed (No data available)","host":"glusterfs-unstable-02.localdomain"} 2013-07-08T00:52:43+09:00 filtered.glusterd {"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"760057","log_level":"I","source_file_name": "socket.c","source_line":"2236","function_name":"socket_event_handler","component":"0transport","message":"disconnecting now","host":"glusterfs-unstable-02.localdomain"} Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   83 
  • 84. Tips 3: Logging Also available on Gluster Forge: https://forge.gluster.org/fluent-plugin-glusterfs Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   84 
  • 85. Tips 3: Logging More detailed explanation in Japanese is here!: http://www.nttpc.co.jp/company/r_and_d/technology/gluster_fs_fluentd.html Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   85 
  • 86. Conclusion Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   86 
  • 87. Conclusion !  Deployment automation tools are useful to install and operate GlusterFS on multiple nodes. !  With cachefilesd, NFS can might be more efficient under heavy read workload. !  Aggregating GlusterFS logs is easy enough and you can utilize them. Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   87 
  • 88. To contact us, e-mail here -> storage-contact@nttpc.co.jp Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   88