gLite Data Management System

Architecture of the gLite Data Management System Leandro Neumann Ciuffo INFN-Catania (Italy) EELA-2 Tutorial Montevideo, 22.07.2009

Outline ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],gLite DMS – EELA-2 Tutorial, 22.07.2009

Challenges ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],gLite DMS – EELA-2 Tutorial, 22.07.2009 Storage Resource Manager interface File Catalogue File Transfer Service Metadata Service

Getting started ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],gLite DMS – EELA-2 Tutorial, 22.07.2009

Getting started ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],gLite DMS – EELA-2 Tutorial, 22.07.2009

Types of Storage Elements ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],gLite DMS – EELA-2 Tutorial, 22.07.2009

Storage Resource Manager (SRM) B C Worker Nodes A User Interface SE - CASTOR SE - DPM dCache submit read input read input store output gLite DMS – EELA-2 Tutorial, 22.07.2009 myJOB

Storage Resource Manager (SRM) ,[object Object],SRM I talk to them on your behalf I will even allocate space for your files And I will use transfer protocols to send your files there SE CASTOR SE DPM SE dCache The SRM is a single interface that takes care of local storage interaction and provides a Grid interface to the outside world. gLite DMS – EELA-2 Tutorial, 22.07.2009

File Naming conventions (1) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],gLite DMS – EELA-2 Tutorial, 22.07.2009 Logical File Name 1 Logical File Name N GUID ...

File Naming conventions (2) ,[object Object],[object Object],[object Object],[object Object],[object Object],Logical File Name 1 Logical File Name N GUID ... ... Physical File SURL N Physical File SURL 1 TURL 1 TURL 1 ... gLite DMS – EELA-2 Tutorial, 22.07.2009

SRM interactions SRM ,[object Object],[object Object],[object Object],[object Object],[object Object],2 3 5 1 4 SE gLite DMS – EELA-2 Tutorial, 22.07.2009 Client

Needles in a haystack ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],gLite DMS – EELA-2 Tutorial, 22.07.2009 File Catalogue

File Catalogue ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],gLite DMS – EELA-2 Tutorial, 22.07.2009

The LFC Service User Interface SE B SE A SE C File Catalogue lfn:/grid/gilda/tcaland/mpi.txt gLite DMS – EELA-2 Tutorial, 22.07.2009

The LFC Service srm://host.example.com/foo/bar host.example.com /grid/dteam/dir1/dir2/file1.root LFN GUID 38ed3f60-c402-11d7 -a6b0… Replicas /grid/dteam/mydir/mylink Symlink Further LFNs can be added as symlinks to the main LFN. LCF key SURLs User Metadata System Metadata gLite DMS – EELA-2 Tutorial, 22.07.2009

Job submission – example 1 User Interface CE Worker Nodes WMS ,[object Object],gLite DMS – EELA-2 Tutorial, 22.07.2009

Data Management – example 2 User Interface CE Worker Nodes WMS LFC SE SE gLite DMS – EELA-2 Tutorial, 22.07.2009

LFC commands ,[object Object],gLite DMS – EELA-2 Tutorial, 22.07.2009 Add/replace a comment lfc-setcomment Set file/directory access control lists lfc-setacl Remove a file/directory lfc-rm Rename a file/directory lfc-rename Create a directory lfc-mkdir List file/directory entries in a directory lfc-ls Make a symbolic link to a file/directory lfc-ln Get file/directory access control lists lfc-getacl Delete the comment associated with the file/directory lfc-delcomment Change owner and group of the LFC file-directory lfc-chown Change access mode of the LFC file/directory lfc-chmod

lcg-utils commands ,[object Object],[object Object],[object Object],gLite DMS – EELA-2 Tutorial, 22.07.2009 lcg-cp Copies a grid file to a local destination lcg-cr Copies a file to a SE and registers the file in the catalog lcg-del Delete one file lcg-rep Replication between SEs and registration of the replica lcg-gt Gets the TURL for a given SURL and transfer protocol lcg-sd Sets file status to “Done” for a given SURL in a SRM request

Environment Variables ,[object Object],[object Object],[object Object],[object Object],[object Object],gLite DMS – EELA-2 Tutorial, 22.07.2009

Let’s practice! Reference: https://grid.ct.infn.it/twiki/bin/view/GILDA/DataManagement

Environment Variables ,[object Object],[object Object],echo $ LCG_GFAL_INFOSYS export LCG_GFAL_INFOSYS =gilda-bdii.ct.infn.it:2170 echo $ LFC_HOST export LFC_HOST =lfc-gilda.ct.infn.it gLite DMS – EELA-2 Tutorial, 22.07.2009

Before starting… voms-proxy-init --voms gilda gLite DMS – EELA-2 Tutorial, 22.07.2009 ,[object Object]

LFC: Listing file and directory lfc-ls -l /grid/gilda ,[object Object],[object Object],Defined by the user LFC Namespace ,[object Object],export LFC_HOME =/grid/gilda/tutorials lfc-ls gLite DMS – EELA-2 Tutorial, 22.07.2009

LFC: creating a directory lfc-mkdir /grid/gilda/tutorials/ yourname ,[object Object],[object Object],[object Object],lfc-ls /grid/gilda/tutorials gLite DMS – EELA-2 Tutorial, 22.07.2009

Downloading a file lcg-cp --vo gilda lfn:/grid/gilda/users/example/alien.txt file://$HOME/alien.txt ,[object Object],[object Object],[object Object],lcg-cp --vo <vo name> <LFN origin> <local destination> gLite DMS – EELA-2 Tutorial, 22.07.2009

Copying and registering a file lcg-cr --vo <vo name> -l <LFN destination> -d <SE> <local file> ,[object Object],[object Object],[object Object],gLite DMS – EELA-2 Tutorial, 22.07.2009 Make sure to have a directory in the LFC ( /grid/gilda/users/sagrid/yourname/ ) Use the lcg-info or lcg-infosites commands to figure out the available SEs lcg-infosites --vo gilda se Avail Space(Kb) Used Space(Kb) Type SEs ---------------------------------------------------------- 1100000000 1145007 n.a gilda-se.rediris.es 1030000000 32 n.a fn2.hpcc.sztaki.hu 295250000 75945624 n.a aliserv6.ct.infn.it n.a 999999 n.a se-edu.grid.acad.bg 60440000 3280565 n.a iceage-se-01.ct.infn.it 1008437 8844236 n.a se.hpc.iit.bme.hu 53160000 440416 n.a vega-se.ct.infn.it 2430000000 440450 n.a se1-egee.srce.hr 97890000 440423 n.a dgt02.ui.savba.sk lcg-cr --vo gilda -l lfn:/grid/gilda/tutorials/ yourname/yourfile.txt -d aliserv6.ct.infn.it file://$HOME/alien.txt

Replicate a file between SEs lcg-rep --vo gilda -d gilda-se.rediris.es lfn:/grid/gilda/tutorials/ yourname/yourfile.txt ,[object Object],[object Object],lcg-rep --vo <vo name> -d <destination SE> <LFN of your file> gLite DMS – EELA-2 Tutorial, 22.07.2009

Listing the replicas ,[object Object],[object Object],[object Object],lcg-lr --vo gilda lfn:/grid/gilda/tutorials/ yourname/yourfile.txt gLite DMS – EELA-2 Tutorial, 22.07.2009

Adding metadata information lfc-setcomment /grid/gilda/tutorials/ yourname/yourfile.txt “ This is my comment ” ,[object Object],[object Object],[object Object],lfc-setcomment <LFC file path> "Your comments" gLite DMS – EELA-2 Tutorial, 22.07.2009

Listing with comments lfc-ls --comment /grid/gilda/tutorials/ yourname/ ,[object Object],gLite DMS – EELA-2 Tutorial, 22.07.2009

Creating a symbolic link ,[object Object],[object Object],[object Object],[object Object],lfc-ln -s /grid/gilda/tutorials/ yourname/yourlink.txt /grid/gilda/tutorials/ yourname/yourfile.txt lfc-ln -s <your symbolic link> <original file> lfc-ls -l /grid/gilda/tutorials/ yourname/ gLite DMS – EELA-2 Tutorial, 22.07.2009

Downloading a file lcg-cp --vo gilda lfn:/grid/gilda/tutorials/ yourname/yourfile.txt file://$HOME/ yourfile.txt ,[object Object],[object Object],lcg-cp --vo <vo name> <LFN origin> <local destination> gLite DMS – EELA-2 Tutorial, 22.07.2009

Deleting a file lcg-del -a --vo gilda lfn:/grid/gilda/tutorials/ yourname/yourfile.txt ,[object Object],[object Object],lcg-del -a --vo <vo name> <LFN> gLite DMS – EELA-2 Tutorial, 22.07.2009

Removing a LFC directory ,[object Object],[object Object],lfc-rm -r <LFC file path> lfc-rm -r /grid/gilda/tutorials/ yourname gLite DMS – EELA-2 Tutorial, 22.07.2009

Get the file SURL ,[object Object],[object Object],[object Object],lcg-lr --vo gilda lfn:/grid/gilda/tutorials/ yourname/yourfile.txt lcg-lr --vo <vo name> <LFN> gLite DMS – EELA-2 Tutorial, 22.07.2009

Get the file TURL lcg-gt <paste the file SURL: srm://…> gsiftp ,[object Object],[object Object],lcg-gt <file SURL> <protocol supported by the SE> gLite DMS – EELA-2 Tutorial, 22.07.2009

gLite Data Management System

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (20)

Semelhante a gLite Data Management System

Semelhante a gLite Data Management System (20)

Mais de Leandro Ciuffo

Mais de Leandro Ciuffo (20)

Último

Último (20)

gLite Data Management System