gLite Data Management File Transfer Agents

1. Release Description

This release contains the gLite Data Management File Transfer Agents module v. 2.1.0. The following sections provide additional information about the release content, the module dependencies, the know bugs and issues and a list of bugs closed since the previous release. For information about installing and using the gLite Data Management File Transfer Agents, please refer to the gLite Installation and User Guides.

2. Changes in this release

2.1. Changes in functionality

The main changes in this release are:

 

• Upgrade to FTS Queue Schema v. 2.2.0: new properties are
  
  - File finishTime: store the time when the file finished/failed
  - File currentFailures: number of failures during current iterations. This value is reset when the FTA move a File from HOLD to PENDING
  - Job submitHost: the hostname of the FTS that received the transfer request
  - Job maxTimeInQueue: (Not yet implemented) number of seconds a job can stay in queue
• Move Retry Action to use Strategy Pattern
• Allow to override Basic Retry Strategy with a python script
• Refactored Unit Tests
• Refactored config
• Refactored dependencies to use libtool .la files
• Drop Custom Actions from the VO Agents, since it wasn't clear how to use them
• Set the default Retry Interval for the VO Agent to 60 seconds instead of using the value provided by the DefaultInterval property (3 seconds)
• Don't try to normalize the TURLs
• Add a smarter RetryStrategy as a python script
• Add python UnitTests for Retry Strategies
• Fix typo in channel-agent configuration parameter VOShareType: values should be normalized and normalized-on-active, not nomalized (Expert parameter, shouldn't affect the deployment modules)
• Exported modules configuration templates
• Added configuration schema version in the generated service config templates
• Improved SD for VO support
• Fixed SRMCopy Transfer status in case of error
• Added myproxy error into the file reason when job fail to get delegated credentials
• Fixed Myproxy certificate caching
• Improved VO share query in the Oracle Connector (using analytics)
• Cleared MyProxy Error Message
• Normalized SURLs
• Cache SD information changes:
  
  - Added transfer-agent-sd.EnableCache expert configuration option to enable/disable the SD Cache (default = true)
  - Added transfer-vo-agent.CleanSDCache_Interval expert configuration option to clean periodically the cache in the VO Agent (default = 3600)
  - Added transfer-channel-agent.CleanSDCache_Interval expert configuration option to clean periodically the cache in the Channel Agent (default = 3600)
  
  
• Refactored a bit glite/data/transfers/agent/module/*.h in order to make the export simpler (this would be required by future versions)
• Added the source and destination endpoint capability setting to glite-url-copy
• Use new VO configuration model with all VO-specific parameters grouped in a single place (vo-list.cfg.xml)

 

2.2. Changes in Configuration

All parameters used to set VO-specific values have been removed from the service configuration files and moved to a dedicated VO configuration file called vo-list.cfg.xml. A template is available in /opt/glite/etc/config/templates. This file is common to all gLite services running on this node or referenced in the same site-config file (unless locally overridden). For more information please refer to the gLite 1.5 installation guide.

 

The R-GMA Service Publisher (aka R-GMA servicetool) parameters are now automatically handled by the configuration script and do not need to be explicitly set anymore in the configuration file. The parameters can be set in the configuration file to override the default values.

 

The Service Discovery parameters for file-based discovery have been removed from the configuration file. File-based discovery is only supported for test or debugging. The normal mode of operation is based on R-GMA or BD-II discovery. If non-gLite services are used by this module, they can be published in R-GMA using the gLite R-GMA Service Publisher tool and the provided templates for external applications.

3. Release contents

The gLite Data Management File Transfer Agents v. 2.1.0 is composed of the following gLite components:

 

Component name	Description	Version	File
glite-config	gLite configuration scripts	1.6.20	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-config-1.6.20-2.noarch.rpm
glite-data-catalog-api-c	gLite Data Catalogs C/C++ API	2.0.0	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-data-catalog-api-c-2.0.0-4.noarch.rpm
glite-data-catalog-cli	Data Catalog CLI	1.7.2	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-data-catalog-cli-1.7.2-4.noarch.rpm
glite-data-catalog-interface	gLite Catalog Interfaces	2.0.0	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-data-catalog-interface-2.0.0-7.noarch.rpm
glite-data-config-service	GLite Data Config Service	1.2.4	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-data-config-service-1.2.4-1.noarch.rpm
glite-data-srm-api-c	gLite Data SRM C/C++ API	1.1.0	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-data-srm-api-c-1.1.0-1.noarch.rpm
glite-data-srm-cli	Data SRM CLI	1.2.3	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-data-srm-cli-1.2.3-1.noarch.rpm
glite-data-transfer-agents	gLite Data Transfer Agents	2.2.0	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-data-transfer-agents-2.2.0-1.noarch.rpm
glite-data-transfer-url-copy	gLite Data Transfer URL Copy	1.5.0	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-data-transfer-url-copy-1.5.0-1.noarch.rpm
glite-data-util-c	Data C util libraries	1.2.0	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-data-util-c-1.2.0-1.noarch.rpm
glite-essentials-cpp	Essential C/C++ libraries for gLite software	1.1.1	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-essentials-cpp-1.1.1-1_EGEE.noarch.rpm
glite-file-transfer-agents-config	gLite File Transfer Agents configuration files	2.1.0	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-file-transfer-agents-config-2.1.0-1.noarch.rpm
glite-rgma-servicetool-config	gLite R-GMA servicetool installation	5.2.2	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-rgma-servicetool-config-5.2.2-1.noarch.rpm
glite-security-utils-config	gLite Security Utilities configuration files	1.2.2	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-security-utils-config-1.2.2-1.noarch.rpm
glite-security-voms-api-c	Change me !!!	1.6.10	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-security-voms-api-c-1.6.10-0.noarch.rpm
glite-service-discovery-api-c	C API for Service Discovery.	2.2.0	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-service-discovery-api-c-2.2.0-0.noarch.rpm
glite-service-discovery-config	gLite Service Discovery installation	1.1.1	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-service-discovery-config-1.1.1-1.noarch.rpm
glite-service-discovery-file-c	File based Service Discovery	2.1.0	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-service-discovery-file-c-2.1.0-2.noarch.rpm

4. Dependencies

The gLite File Transfer Agents module v. 2.1.0 has the following dependencies:

 

Component name	Description	Version	File
CGSI_gSOAP_2.6	GSI plugin for gSOAP	1.1.9	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/CGSI_gSOAP_2.6-1.1.9-1.slc3.noarch.rpm
gpt	The Grid Packaging Toolkit (GPT)	VDT1.2.2rh9	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/gpt-VDT1.2.2rh9-1.noarch.rpm
myproxy	Credential repository for the Grid.	1.14	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/myproxy-1.14-EGEE.noarch.rpm
perl-Expect.pm	Expect.pm module for perl	1.01	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/perl-Expect.pm-1.01-9.noarch.rpm
vdt_globus_essentials	Virtual Data Toolkit	VDT1.2.2rh9	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/vdt_globus_essentials-VDT1.2.2rh9-1.noarch.rpm

 

The Oracle InstantClient libraries and executables are not explicitly added as dependencies in the RPMS, but they have to be installed before using the File Transfer Agents with an Oracle backend. The Oracle Instantclient is not freely redistributable and is not included with this release. The required RPMS or tarballs have to be downloaded from the Oracle web site and installed on the FTA node.

 

5. Known bugs and issues

This release has the following known bugs and issues. Bug numbers refer to the gLite Bug Tracking system database hosted on the CERN Savannah system at https://savannah.cern.ch/bugs/?group=jra1mdw .

 

• FTA doesn't respect the transfer limit imposed in the channel table. When starting new transfers, the channel agent sometime submit more file than the limit configured in the channel table
• FTS: Inconsistent jobs' state in FTA
• Configuration script fails on an existing 1.4.1 FTA configuration file. The configuration file has to be manually modified to follow the new configuration schema
• No removal procedure is provided with this release apart from the removal of the RPMS. Any account, group or other resource created during the module configuration must be manually cleaned.

 

Known open bugs:

 

Bug number	Description
#12218	In the template glite-file-transfer-agents.cfg.xml a secure url is shown to start with http and not https
#12416	1.4 FTA documentation is not complete
#12508	1.4 FTA documentation is not complete when mentioning the glite-transfer-channel-setvoshare step
#12603	1.4 FTA configuration script fails to configure the proxy server details
#13466	file-transfer-agents config script seems to have some problems generating glite_setenv.(c)sh
#13825	FTS: FTA: agent configuration script -h option says --remove remove schema when it doesn't
#14272	FTS Config Script: script fails on an existing 1.4.1 FTA configuration file

 

Bugs fixed in this or previous releases but not yet tested:

 

Bug number	Description
#9793	FTS: FTA wrong default value for vo-agent
#10060	FTS: FTA 1.3: does not retry submission after the retry interval time
#10317	FTS: FTA installation script doesn't allow to ovewrite parameters defined in glite-global.cfg.xml
#10329	FTS: FTA doesn't retry transfers when urlcopy doesn't return a reason class
#10490	FTS: FTA channel agent with MySQL backend don't start
#10534	FTS: FTA MySQL: agents get the wrong type
#12602	FTS: 1.4 FTA does not work from castor to castor with srmcopy
#12961	FTS: FTA uses the wrong service type for GridFTP services
#13301	FTS: FTA doesn't respect the transfer limit imposed in the channel table
#13724	FTS: Inconsistent jobs' state in FTA

 

 

6. Bugs closed since last release

This release fixes the following bugs and issues. Bug numbers refer to the gLite Bug Tracking system database hosted on the CERN Savannah system at https://savannah.cern.ch/bugs/?group=jra1mdw  

 

Bug number	Description
#9803	FTS: FTA logrotate does not rotate logs
#9830	FTS: FTA log file names could be improved for clarity
#9913	Oracle instantclient RPM deps should be removed from all modules
#9944	FTS: FTA logrotate is not configured
#9946	FTS: FTA+FTS configuration on the same node does not work using site config files
#9981	Installer broken for 1.3 glite-data-transfer-agents-oracle_installer.sh
#10055	FTS: FTA global user parameter file-transfer.VONAME should be remove
#10582	FTS: FTA SRM copy log file are way too verbose...
#10584	FTA: FTA SRM agent tries a finish request even the SRM copy has failed.
#10588	FTS: FTA canceled job does not want to die...
#10720	FTS: FTA. SRM copy: not all requests are filled with duration