Description

GFAL 2 utils are a group of command line tools for file manipulations with any protocol managed by gfal2

Installation

yum install gfal2-util gfal2-all

List of command line tools

gfal-cat Concatenate a list of files into the standard output.

gfal-chmod Change permissions for a file or directory.

gfal-copy Copy files from source to destination(s).

gfal-legacy-bringonline Require a tape system to stage a file.

gfal-legacy-register Register a new replica into a catalog.

gfal-legacy-replicas List the replicas of an entry in a catalog.

gfal-legacy-unregister Remove a replica from a catalog.

gfal-ls Equivalent to the system `ls` command.

gfal-mkdir Create a directory.

gfal-rename Rename a file or directory.

gfal-rm Remove a file or directory (with -r).

gfal-save Write standard input to a file.

gfal-stat Show extended information about a file or directory.

gfal-sum Retrieve the file checksum.

gfal-xattr List or display a file's extended attributes.

Third party copy

To understand some parts of this documentation, it is useful to know the difference between a third party copy and a regular one.

Third party copy: When doing a copy between two remote endpoints, the data is sent directly between the two participating storages.
Non-third party copy: The data goes through the client doing the copy.

Gfal2 is capable of doing both, depending on the protocol combination. Some protocols do support third party copy (see below), some others do not. In any case, when copying between two different protocols, the copy will not be third party copy (e.g from GridFTP to DAV).
SRM is an exception to this.

Protocol support

Gfal2 relies on plugins to add support for different protocols. Depending on your installation, some may not be available.
To use a given protocol, the general syntax is protocol://host[:port]/path. For instance:

• gsiftp://dpm.cern.ch/dpm/cern.ch/home/dteam
• davs://dpm.cern.ch/dpm/cern.ch/home/dteam
• davs://dpm.cern.ch:443/dpm/cern.ch/home/dteam

SRM is an exception to this.

You can check what is installed using rpm -qa | grep gfal2-plugin. The normally available protocols are:

file://

Local file access. It can be omitted (i.e. gfal-ls /).

gsiftp://

GridFTP protocol. It has third-party copy support.

srm://

SRM protocol. Its support for third-party copies depends on the transfer URL negotiated with the endpoint, and the protocol used for the other peer.
gfal2 is capable of resolving the full SURL using the information system (e.g srm://host/path), but otherwise, you can also use the full SURL (e.g srm://host:8446/srm/managerv2?SFN=/path)

root://

XRootD protocol. Some storages support third party copies, others do not.

dav(s)://

WebDAV protocol. Some storages support third party copies, but to enable it, you need to use davs+3rd://

https://

Plain HTTPS protocol. Some storages support third party copies, but to enable it, you need to use https+3rd://

s3://

S3 protocol. Does not support third party copy, but it could still be achieved if the peer is a davs+3rd endpoint

lfc://

LFC endpoints. Requires the endpoint to be part of the url (lfc://lfc.cern.ch/path)

guid:

LFC endpoints, by GUID. It uses the LFC_HOST configured under /etc/gfal2.d/lfc_plugin.conf

rfio://

RFIO protocol.

dcap://

DCAP protocol, only supported by dCache endpoints.

These plugins may not be normally installed.

sftp://

Experimental. SFTP support. Works with an SSH private key, instead of a X509 proxy.

mock://

Useful for testing. Allows to force a given reply depending on the URL used. See its documentation.

Configuration

All plugin configuration files are stored under /etc/gfal2.d/. You can navigate them and have a look at the different configuration options on the gfal2 repository.

In any case, for all commands, any value can be overridden by using the flag -D"GROUP:PARAM=VALUE"

For example, let's say we want to do a listing of an SRM endpoint, but using a different infosys endpoint.

gfal-ls -D"BDII:LCG_GFAL_INFOSYS=myinfosys.cern.ch:2170" srm://endpoint.cern.ch/dpm/cern.ch/home/dteam/

Examples

Uploading a file to the Grid

gfal-copy -fpK /home/doe/file1 srm://dpmhead-rc.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/dir/nested/file1

In this example, the flag -f tells gfal-copy to overwrite the destination if it already exists, -p enables automatic directory creation, and -K enables checksum validation.

Copying files out of the Grid

gfal-copy can be used for uploading, downloading, and also third party copies. Same syntax applies.

gfal-copy -fpK srm://dpmhead-rc.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/dir/nested/file1 /home/doe/file1

Listing files and directories

gfal-ls -l srm://dpmhead-rc.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/
-rw-rw-r--   1 46    45          192 Jun 21  2016 853.test
-rw-rw-r--   1 46    45       641064 Jun 16  2016 aaaabbaaa
-rw-rw-r--   1 46    48            0 Jun  6  2016 build.out
-rw-rw-r--   1 46    49        49185 Nov 18 16:04 dmc-901.test
-rw-rw-r--   1 46    49        49185 Nov 18 16:05 dmc-902.test
-rw-rw-r--   1 46    50      5279801 Apr 28 16:20 test

In this example, -l enables long listing, which displays then mode, owner, size and mtime next to each entry. Without -l, the short version is displayed instead: only the names.

gfal-ls uses the environment variable LS_COLORS when -l is passed, if the terminal supports it.

Get information about a file or directory

gfal-stat srm://dpmhead-rc.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/
  File: 'srm://dpmhead-trunk.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch'
  Size: 0	directory
Access: (0755/drwxr-xr-x)	Uid: 45	Gid: 45
Access: 1970-01-01 01:00:00.000000
Modify: 2015-06-10 18:26:00.000000
Change: 2017-05-09 11:42:52.000000

Get the checksum of a file

gfal-sum srm://dpmhead-trunk.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/aalvarez/aaaabbaaa adler32
srm://dpmhead-trunk.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/aalvarez/aaaabbaaa c88e70d6

gfal-sum srm://dpmhead-trunk.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/aalvarez/aaaabbaaa md5
srm://dpmhead-trunk.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/aalvarez/aaaabbaaa 64fd36ae477e970b06949fe90d0c8c20

Get information about space tokens

gfal-xattr srm://dpmhead-trunk.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/ spacetoken
["65e3ad04-ada1-11e6-b531-c860001bd938","bd1dae50-108c-11e7-82fa-c860001bd938","77056126-194f-11e7-82fa-c860001bd938","4ad03b22-2a92-11e7-82fa-c860001bd938","45e35c78-2b52-11e7-82fa-c860001bd938"]

gfal-xattr srm://dpmhead-trunk.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/ spacetoken.token?65e3ad04-ada1-11e6-b531-c860001bd938
{ "spacetoken": "65e3ad04-ada1-11e6-b531-c860001bd938", "owner": "root", "totalsize": 14937625, "unusedsize": 112193261767, "usedsize": 0, "guaranteedsize": 107374182400, "lifetimeleft": -1, "retention": "CUSTODIAL", "accesslatency": "NEARLINE" }

Apendix: lcg-util equivalence table

(*) These actions are actually performed internally by the SRM plugin when needed.

License

This software is licensed under the Apache 2 License. See LICENSE for details.

Release Notes

See RELEASE-NOTES for a detailed changelog.

Contact

You can notify bugs or ask for feature requests via

Tracker

https://its.cern.ch/jira/browse/DMC/component/12918

Mailing list

dmc-support@cern.ch