Description
GFAL 2 utils are a group of command line tools for file manipulations with any protocol managed by gfal2
Installation
yum install gfal2-util gfal2-all
List of command line tools
gfal-cat Concatenate a list of files into the standard output.
gfal-chmod Change permissions for a file or directory.
gfal-copy Copy files from source to destination(s).
gfal-legacy-bringonline Require a tape system to stage a file.
gfal-legacy-register Register a new replica into a catalog.
gfal-legacy-replicas List the replicas of an entry in a catalog.
gfal-legacy-unregister Remove a replica from a catalog.
gfal-ls Equivalent to the system `ls` command.
gfal-mkdir Create a directory.
gfal-rename Rename a file or directory.
gfal-rm Remove a file or directory (with -r).
gfal-save Write standard input to a file.
gfal-stat Show extended information about a file or directory.
gfal-sum Retrieve the file checksum.
gfal-xattr List or display a file's extended attributes.
Third party copy
To understand some parts of this documentation, it is useful to know the difference between a third party copy and a regular one.
Third party copy: When doing a copy between two remote endpoints, the data is sent directly between the two participating storages.
Non-third party copy: The data goes through the client doing the copy.
Gfal2 is capable of doing both, depending on the protocol combination. Some protocols do support third party copy (see below), some others do not. In any case, when copying between two different protocols, the copy will not be third party copy (e.g from GridFTP to DAV).
SRM is an exception to this.
Protocol support
Gfal2 relies on plugins to add support for different protocols. Depending on your installation, some may not be available.
To use a given protocol, the general syntax is protocol://host[:port]/path. For instance:
- gsiftp://dpm.cern.ch/dpm/cern.ch/home/dteam
- davs://dpm.cern.ch/dpm/cern.ch/home/dteam
- davs://dpm.cern.ch:443/dpm/cern.ch/home/dteam
SRM is an exception to this.
You can check what is installed using rpm -qa | grep gfal2-plugin. The normally available protocols are:
file://
Local file access. It can be omitted (i.e. gfal-ls /).
gsiftp://
GridFTP protocol. It has third-party copy support.
srm://
SRM protocol. Its support for third-party copies depends on the transfer URL negotiated with the endpoint, and the protocol used for the other peer.
gfal2 is capable of resolving the full SURL using the information system (e.g srm://host/path), but otherwise, you can also use the full SURL (e.g srm://host:8446/srm/managerv2?SFN=/path)
root://
XRootD protocol. Some storages support third party copies, others do not.
dav(s)://
WebDAV protocol. Some storages support third party copies, but to enable it, you need to use davs+3rd://
https://
Plain HTTPS protocol. Some storages support third party copies, but to enable it, you need to use https+3rd://
s3://
S3 protocol. Does not support third party copy, but it could still be achieved if the peer is a davs+3rd endpoint
lfc://
LFC endpoints. Requires the endpoint to be part of the url (lfc://lfc.cern.ch/path)
guid:
LFC endpoints, by GUID. It uses the LFC_HOST configured under /etc/gfal2.d/lfc_plugin.conf
rfio://
RFIO protocol.
dcap://
DCAP protocol, only supported by dCache endpoints.
These plugins may not be normally installed.
sftp://
Experimental. SFTP support. Works with an SSH private key, instead of a X509 proxy.
mock://
Useful for testing. Allows to force a given reply depending on the URL used. See its documentation.
Configuration
All plugin configuration files are stored under /etc/gfal2.d/. You can navigate them and have a look at the different configuration options on the gfal2 repository.
In any case, for all commands, any value can be overridden by using the flag -D"GROUP:PARAM=VALUE"
For example, let's say we want to do a listing of an SRM endpoint, but using a different infosys endpoint.
gfal-ls -D"BDII:LCG_GFAL_INFOSYS=myinfosys.cern.ch:2170" srm://endpoint.cern.ch/dpm/cern.ch/home/dteam/
Examples
Uploading a file to the Grid
gfal-copy -fpK /home/doe/file1 srm://dpmhead-rc.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/dir/nested/file1
In this example, the flag -f tells gfal-copy to overwrite the destination if it already exists, -p enables automatic directory creation, and -K enables checksum validation.
Copying files out of the Grid
gfal-copy can be used for uploading, downloading, and also third party copies. Same syntax applies.
gfal-copy -fpK srm://dpmhead-rc.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/dir/nested/file1 /home/doe/file1
Listing files and directories
gfal-ls -l srm://dpmhead-rc.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/ -rw-rw-r-- 1 46 45 192 Jun 21 2016 853.test -rw-rw-r-- 1 46 45 641064 Jun 16 2016 aaaabbaaa -rw-rw-r-- 1 46 48 0 Jun 6 2016 build.out -rw-rw-r-- 1 46 49 49185 Nov 18 16:04 dmc-901.test -rw-rw-r-- 1 46 49 49185 Nov 18 16:05 dmc-902.test -rw-rw-r-- 1 46 50 5279801 Apr 28 16:20 test
In this example, -l enables long listing, which displays then mode, owner, size and mtime next to each entry. Without -l, the short version is displayed instead: only the names.
gfal-ls uses the environment variable LS_COLORS when -l is passed, if the terminal supports it.
Get information about a file or directory
gfal-stat srm://dpmhead-rc.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/ File: 'srm://dpmhead-trunk.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch' Size: 0 directory Access: (0755/drwxr-xr-x) Uid: 45 Gid: 45 Access: 1970-01-01 01:00:00.000000 Modify: 2015-06-10 18:26:00.000000 Change: 2017-05-09 11:42:52.000000
Get the checksum of a file
gfal-sum srm://dpmhead-trunk.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/aalvarez/aaaabbaaa adler32 srm://dpmhead-trunk.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/aalvarez/aaaabbaaa c88e70d6
gfal-sum srm://dpmhead-trunk.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/aalvarez/aaaabbaaa md5 srm://dpmhead-trunk.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/aalvarez/aaaabbaaa 64fd36ae477e970b06949fe90d0c8c20
Get information about space tokens
gfal-xattr srm://dpmhead-trunk.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/ spacetoken ["65e3ad04-ada1-11e6-b531-c860001bd938","bd1dae50-108c-11e7-82fa-c860001bd938","77056126-194f-11e7-82fa-c860001bd938","4ad03b22-2a92-11e7-82fa-c860001bd938","45e35c78-2b52-11e7-82fa-c860001bd938"]
gfal-xattr srm://dpmhead-trunk.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/ spacetoken.token?65e3ad04-ada1-11e6-b531-c860001bd938 { "spacetoken": "65e3ad04-ada1-11e6-b531-c860001bd938", "owner": "root", "totalsize": 14937625, "unusedsize": 112193261767, "usedsize": 0, "guaranteedsize": 107374182400, "lifetimeleft": -1, "retention": "CUSTODIAL", "accesslatency": "NEARLINE" }
Apendix: lcg-util equivalence table
Description | lcg-util | gfal-util |
---|---|---|
Add an alias for a given GUID | lcg-aa | Deprecated |
Bring SURLs online | lcg-bringonline | gfal-legacy-bringonline |
Copy files with no catalog involved | lcg-cp | gfal-copy |
Copy and register a file | lcg-cr | gfal-copy |
Delete a file / directory | lcg-del | gfal-rm |
Get the file checksum | lcg-get-checksum | gfal-sum |
Get the TURLs for given SURLs | lcg-getturls | gfal-xattr "user.replicas" |
Get the TURL for a given SURL | lcg-gt | gfal-xattr "user.replicas" |
List aliases for a given LFN/GUID | lcg-la | Deprecated |
Get the GUID for a given LFN | lcg-lg | gfal-xattr "user.guid" |
Lists the replicas for a given LFN | lcg-lr | gfal-xattr "user.replicas" |
List information / directory | lcg-ls | gfal-ls |
Remove an alias | lcg-ra | Deprecated |
Copy between Ses using the catalog | lcg-rep | gfal-copy |
Register a file in a catalog | lcg-rf | gfal-copy |
Set a file status to done | lcg-sd | Deprecated* |
Get space tokens associated to a description | lcg-stmd | gfal-xattr "spacetoken"/"spacetoken.token?$tok"/"spacetoken.description?$desc" |
Unregister a file | lcg-uf | gfal-legacy-unregister |
Create a directory | gfal-mkdir | |
Dump the standard input into a file | gfal-save | |
Dump a file into the standard output | gfal-cat |
(*) These actions are actually performed internally by the SRM plugin when needed.
License
This software is licensed under the Apache 2 License. See LICENSE for details.
Release Notes
See RELEASE-NOTES for a detailed changelog.
Contact
You can notify bugs or ask for feature requests via
Tracker
https://its.cern.ch/jira/browse/DMC/component/12918
Mailing list