gLite Data Management File Transfer Service

1. Release Description

This release contains the gLite Data Management File Transfer Service module v. 3.0.0. The following sections provide additional information about the release content, the module dependencies, the know bugs and issues and a list of bugs closed since the previous release. For information about installing and using the gLite Data Management File Transfer Service, please refer to the gLite Installation and User Guides.

2. Changes in this Release

The main changes in this release are:

This version of the File Transfer Service module supports both Oracle and MySQL backends
The interface was changed: the FPS interface was merged into the FTS interface, but FTS stayed backward compatible with R1.3.
There is a new FileTransferStats interface to ease the query of statistical information of the transfers
Sync with the interface changes and added a new functionality to list the running agents. Job submission now checks if there is an agent, which will be able to process the transfer.
There were schema chandes due to the agent check and statistics functionalities, but there is a schema upgrade script provided with the deployment module that also removes the old web application and migrate existing configuration files.
There is also an R1.1.2 compatibility endpoint provided for old clients (the R1.1.2->R1.3 was not a backward compatible change).

3. Release contents

The gLite Data Management File Transfer Service v. 3.0.0 is composed of the following gLite components:

Component name	Description	Version	File
glite-config	gLite configuration scripts	1.4.2	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/noarch/RPMS/glite-config-1.4.2-1.noarch.rpm
glite-data-catalog-api-java	GLite Catalog Services Java client	2.0.0	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/noarch/RPMS/glite-data-catalog-api-java-2.0.0-6.noarch.rpm
glite-data-common	common files for gLite Data Management web-services	1.3.0	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/noarch/RPMS/glite-data-common-1.3.0-1.noarch.rpm
glite-data-transfer-api-c	GLite Data Transfer C/C++ API	3.1.0	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/i386/RPMS/glite-data-transfer-api-c-3.1.0-1.i386.rpm
glite-data-transfer-cli	Data Transfer CLI	3.1.0	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/i386/RPMS/glite-data-transfer-cli-3.1.0-3.i386.rpm
glite-data-transfer-fts	gLite File Transfer Service (FTS)	3.1.0	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/noarch/RPMS/glite-data-transfer-fts-3.1.0-9.noarch.rpm
glite-data-transfer-interface	gLite Transfer Interfaces	3.1.0	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/noarch/RPMS/glite-data-transfer-interface-3.1.0-3.noarch.rpm
glite-data-util-c	Data C util libraries	1.1.4	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/i386/RPMS/glite-data-util-c-1.1.4-1.i386.rpm
glite-file-transfer-service-client-config	gLite File Transfer Service client installation	1.1.0	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/noarch/RPMS/glite-file-transfer-service-client-config-1.1.0-1.noarch.rpm
glite-file-transfer-service-config		3.0.0	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/noarch/RPMS/glite-file-transfer-service-config-3.0.0-1.noarch.rpm
glite-rgma-api-c	C API for R-GMA	4.1.13	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/i386/RPMS/glite-rgma-api-c-4.1.13-1.i386.rpm
glite-rgma-api-cpp	C++ API for R-GMA	4.1.15	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/i386/RPMS/glite-rgma-api-cpp-4.1.15-1.i386.rpm
glite-rgma-api-java	Java API for R-GMA	4.1.5	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/noarch/RPMS/glite-rgma-api-java-4.1.5-1.noarch.rpm
glite-rgma-base	R-GMA basic configuration and documentation	4.1.19	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/noarch/RPMS/glite-rgma-base-4.1.19-1.noarch.rpm
glite-rgma-common-config	gLite rgma common configuration items installation	5.0.0	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/noarch/RPMS/glite-rgma-common-config-5.0.0-1.noarch.rpm
glite-rgma-servicetool	R-GMA service tool	4.1.19	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/noarch/RPMS/glite-rgma-servicetool-4.1.19-3.noarch.rpm
glite-rgma-servicetool-config	gLite R-GMA servicetool installation	5.1.0	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/noarch/RPMS/glite-rgma-servicetool-config-5.1.0-1.noarch.rpm
glite-rgma-stubs-servlet-java	Java client implementation stubs for R-GMA	4.1.13	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/noarch/RPMS/glite-rgma-stubs-servlet-java-4.1.13-1.noarch.rpm
glite-security-trustmanager	The java certificate path checkin for proxy certs in SSL with plugins for tomcat and axis.	1.7.3	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/noarch/RPMS/glite-security-trustmanager-1.7.3-1.noarch.rpm
glite-security-util-java	The java utilities library for security	1.1.2	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/noarch/RPMS/glite-security-util-java-1.1.2-2.noarch.rpm
glite-security-utils-config	gLite Security Utilities configuration files	1.0.4	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/noarch/RPMS/glite-security-utils-config-1.0.4-1.noarch.rpm
glite-security-voms-api-c		1.6.3	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/i386/RPMS/glite-security-voms-api-c-1.6.3-0.i386.rpm
glite-service-discovery-api-c	C API for Service Discovery.	2.0.1	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/i386/RPMS/glite-service-discovery-api-c-2.0.1-0.i386.rpm
glite-service-discovery-api-java		2.0.1	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/noarch/RPMS/glite-service-discovery-api-java-2.0.1-1.noarch.rpm
glite-service-discovery-bdii-c	BDII based Service Discovery	2.0.2	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/i386/RPMS/glite-service-discovery-bdii-c-2.0.2-1.i386.rpm
glite-service-discovery-cli	Service Discovery CLI	2.0.0	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/i386/RPMS/glite-service-discovery-cli-2.0.0-1.i386.rpm
glite-service-discovery-config	gLite Service Discovery installation	1.1.0	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/noarch/RPMS/glite-service-discovery-config-1.1.0-1.noarch.rpm
glite-service-discovery-file-c	File based Service Discovery	2.0.0	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/i386/RPMS/glite-service-discovery-file-c-2.0.0-1.i386.rpm
glite-service-discovery-rgma-c	C API for Service Discovery.	2.0.2	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/i386/RPMS/glite-service-discovery-rgma-c-2.0.2-0.i386.rpm
glite-service-discovery-rgma-java		2.0.2	../../../../../../../glite-web/egee/packages/R1.4/R20050916/bin/rhel30/noarch/RPMS/glite-service-discovery-rgma-java-2.0.2-1.noarch.rpm

4. Dependencies

The gLite File Transfer Service module v. 3.0.0 has the following dependencies:

Component name	Version	RPM file name
gLite C/C++ Essentials	1.1.1	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/glite-essentials-cpp-1.1.1-1_EGEE.i386.rpm
GPT	VDT 1.2.2	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/gpt-VDT1.2.2rh9-1.i386.rpm
VDT Globus Essentials	VDT 1.2.2	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/vdt_globus_essentials-VDT1.2.2rh9-1.i386.rpm
CGSI_gSOAP_2.6	1.1.8	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/CGSI_gSOAP_2.6-1.1.8-2.slc3.i386.rpm
perl-Expect.pm	1.01	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/perl-Expect.pm-1.01-9.i386.rpm
myproxy	1.14	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/myproxy-1.14-EGEE.i386.rpm
Tomcat	5.0.28	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/tomcat5-5.0.28-10_EGEE.i386.rpm
Java JRE/SDK	1.4.2	http://java.sun.com/j2se/1.4.2/download.html
Oracle InstantClient	>= 10.1.0.3	http://www.oracle.com/technology/software/tech/oci/instantclient/htdocs/linuxsoft.html

The Oracle InstantClient libraries and executables are not freely redistributable and are not included with this release. The four RPMS have to be downloaded from the Oracle web site and installed on the system before installing the gLite File Transfer Service module.

5. Known bugs and issues

This release has the following known bugs and issues. Bug numbers refer to the gLite Bug Tracking system database hosted on the CERN Savannah system at https://savannah.cern.ch/bugs/?group=jra1mdw .

If upgrading from gLite 1.3, the migration script doesn’t remove the VO name from the endpoints published to R-GMA. Since the FTS is not anymore per VO in gLite 1.4, the endpoints should be modified manually after running the update script by removing the VO name from the URL
The gLite I/O and gLite FTS security models conflict. This is being discussed in several coordination bodies. A proposal to harmonize them has been already produced and circulated as part of the updated gLite Architecture document
No removal procedure is provided with this release apart from the removal of the RPMS. Any account, group or other resource created during the module configuration must be manually cleaned.

Known open bugs:

Bug number	Description
#7353	FTS: if XML config file is badly formatted, the glite_config_service tool hangs on start
#7502	FTS: Channel management HALTED state should be set on repeated failures
#8345	gLite I/O and gLite FTS security models conflict
#8364	FTS: Accept not fully qualified SURLs on the server side
#8367	FTS: Handling of full stage pools not optimal
#8368	FTS: Improve retry and channel halting mechanisms
#8369	FTS: Validation of FTS for DB deployment
#8374	FTS: Usage of SRM Copy
#8377	FTS: Trace facility
#8378	FTS: Cleanup of jobs in terminal states in the database
#8380	FTS: Interaction with network monitoring
#8381	FTS: Alarm script to notice mysterious drop in throughput
#8383	FTS: Knowledge of the service portal needed
#8385	FTS: Multi hop means no async transfers
#8587	FTS: (and others) Oracle connections can go stale
#8714	FTS: should be able to specify log4j file in config
#8906	FTS: auto-generated quattor template has a couple of bugs
#8949	FTS: max number of tomcat DB connections in pool should match DB deployment maximum
#9165	FTS: WEB UI does not show all possible file states
#9203	FTS: Would be useful to have a view of ongoing transfers
#9222	FTS: Logrotate script should not print on STDOUT
#9411	FTS: glite-file-transfer-service-oracle-config.py crash if directory is missing
#9499	FTS - Client ignores invalid host certificate.
#9787	FTS: submitting a transfer wihout a proxy gives a SOAP exception
#9834	FTS: good to have backwards compatible interface ports on FTS
#9938	FTS: BDII generic information needed (SD is not sufficient).
#9939	FTS: Need to finish BDII integration
#9959	FTS: scripts to check SRM timings, etc
#9967	FTS: drop Schema for old versions missing in rpm
#9968	FTS: the deployment script does not detect if an incompatible schema version is present
#10076	FTS: should return error immediately if host is not SRM
#10191	FTS: would be good to get rid of count in DB
#10192	FTS: would be nice to have state triggers
#10198	FTS: Should not be possible to open more than one channel between 2 sites
#10200	FTS: dropping of a channel with jobs should not give internal server error
#10221	FTS: schema load file should not drop the existing schema
#10249	FTS: 1.3: daemon segfaults in TransferFile::init
#10479	FTS: cleanlog script does completed when asked to do failed
#10633	FTS: myproxy should be an "associated service" for a FTS
#10634	FTS: FTS must publish which channels it serves

Bugs fixed in this or previous releases but not yet tested:

Bug number	Description
#7354	FTS: glite_config_service waits for all child transfer processes to finish before stopping
#7503	FTS: Add TURL:TURL to DB
#7516	FTS: log rotation on service
#8333	FTS: service is not able to parse correctly the "grid-mapfile" with the administrators of the service
#8371	FTS: Usage of MySQL
#8372	FTS: Security enhancements for the portal
#8379	FTS: MyProxy version is not compatible with the VDT version on LCG-2
#8589	FTS: transfer-submit -f FILE doesn't recognize newline as delimiter
#8713	FTS: when FTS is misconfigured, the client gets obscure error messages
#8911	FTS: often the number of bytes transferred is incorrect in the DB
#8915	FTS: fts starting script reports success when it fails
#8916	FTS: getVersion gives strange error
#8958	FTS: If a mapfile is not specified in the config, a client will receive an InternalException from the service
#8986	FTS: Should build C/C++ code with debug symbols and RPM should not strip the symbols from the code
#9032	FTS: primary timeout doesn't work on FTS url-copy when SRM fails on putDone
#9166	FTS: No simple way to list all requestID's in a state
#9181	FTS: service restart does not always restart FTS agent
#9200	FTS: glite-url-copy uses both jar and tar
#9201	FTS: glite-url-copy-cleanlog usage message issues
#9223	FTS: glite-url-copy print out perf markers in blocking mode?
#9224	FTS: Can't copy large files to dCache
#9232	FTS: glite-url-copy doesn't timeout if file never transitions from pending
#9240	FTS: glite-url-copy does not distinguish in FINAL message on timeouts
#9261	FTS: summary of what hapenning on a channel
#9309	FTS: dCache returns error on gridftp call, but glite-url-copy doesn't see it?
#9310	FTS: two FINAL messages if advisoryDelete fails against dCache
#9322	FTS: Formatting could be better on new log messages
#9338	FTS: agent shoud fill in DN field when it updates a table
#9398	FTS: mysterious failure on PUT when it apparently completed OK
#9399	FTS: blocking glite-url-copy segfaults or buffer overrruns sometimes on srm-putdone error
#9480	FTS: running changeStateForHeldJobs("Failed") corrupts jobstate
#9485	FTS: job/file state corruption in DB
#9500	FTS: FTS daemon lockfile prevents a restart from watchdog script
#9503	FTS: 1.3: gsiftp - gsiftp transfer fails
#9531	FTS: 'reason' buffer is only 1024 bytes, and if overfilled results in Oracle error on update
#9549	FTS: actually use the numbrer of streams per channel setting
#9712	FTS: when myproxy retrieval fails, the DB connection sometimes drops
#9732	FTS: should periodically refresh its mapfiles
#9789	FTS: submit command always prompts for the myproxy password
#9793	FTS: FTA wrong default value for vo-agent
#9813	FTS: 1.3: url-copy: SRM get failure seg faults.
#9826	FTS: org.glite.data: Error while reading directory null
#9833	FTS: It'd be clearer for deployment to have FTS hostcert in /etc/grid-security/
#9835	FTS: VETO mapfile entries allow channel-add but not channel-list
#9839	FTS: should probably switch VETOed people to include Admins
#9883	FPS: the interface has to be able to accept FTS jobs.
#9944	FTS: FTA logrotate is not configured
#9950	FTS: FTS deployment script --status option does not work
#10119	FTS: Full table scan on DB is limiting the throughput rate
#10201	FTS: glite-transfer-channel-add usage needs to be updated
#10222	FTS: DB connection should be per instance
#10235	FTS: must be able to stop individual instances without stopping the others
#10440	FTS: ChannelAgent doesn't always use client delegated credentials for srmcopy
#10490	FTS: FTA channel agent with MySQL backend don't start
#10498	FTS: FTA missing MySQL port FTA configuration parameter
#10534	FTS: FTA MySQL: agents get the wrong type
#10557	FTS: transfers with SRM copy fail

6. Bugs closed since last release

This release fixes the following bugs and issues. Bug numbers refer to the gLite Bug Tracking system database hosted on the CERN Savannah system at https://savannah.cern.ch/bugs/?group=jra1mdw

Bug number	Description
#6931	FTS: Should use user proxy for transfer
#6932	FTS: Should support Oracle for its back end database
#6933	FTS: Should support Channel management
#7113	FTS: glite-url-copy - if SRM.put fails, the code loops forever
#7318	FTS: Transfer agent sometimes mis-counts the number of active transfers
#7352	FTS: forked glite-url-copy should chdir somewhere it can write
#7355	FTS: transfer-url-copy sometimes dies on gridFTP
#7358	FTS: channel allocation fails if the SURL doesn't contain a port number
#7361	FTS: transfer agent should detect a bad Oracle connection, and try to reconnect
#7466	FTS: transfer-url-copy returns 0 for size and bytes written
#7506	FTS: requirement 1.2 should be met: proper retry
#7508	FTS: now needs to implement the protocol option passing as defined in interface
#7517	FTS: rename FilePlacement ot FileTransfer, fps -> fts
#7568	FTS: url-copy sometimes core dumps if SURL format is bad
#7581	FTS: add support for error categories
#7594	FTS: should migrate all components to FileTransfer interface
#7595	FTS: Need to test installation of -fts module
#7596	FTS: Commandline tools should be tested fully against service
#7738	FTS: clean up SRM after transfer failure: advisory delete
#7739	FTS: time out on Gridftp if it hangs on network failure
#7740	FTS: glite-url-copy dumps core on some SRM failures
#8336	FTS: testing methods do not work
#8365	FTS: Command Lines need service resolution from BDII
#8366	FTS: Transfer Agent for a channel is per-VO
#8373	FTS: Certification with dCache, DPM and new castor SRM
#8375	FTS: Daemon watchdog scripts
#8376	FTS: per transfer logs need better management
#8382	FTS: Split deployment model
#8386	FTS: No inter-VO scheduling
#8485	FTS: Deployment script should not set X509_USER_CERT/KEY
#8486	FTS: submit commandline doesn't accept myproxy password
#8503	FTS: detailed transfer-status commandline doesn't work correctly
#8504	FTS: failure message for channel-list could be better
#8538	FTS: channel-set -S Active does not work.
#8592	FTS: command-line and java service mispelling of Canceling and Canceled
#8670	FTS: SC3 Quattor template has missing deps
#8690	FTS: channel management setStatusForHeldJobs has problems
#8701	FTS: server should accept equalsIgnoreCase("gridftp") for the transfer params
#8702	FTS: default portal logging should be DEBUG
#8707	FTS: Deployment script should be update in light of discussions with DB people
#8708	FTS: transfer-cli tools should not depend on catalog-cli tools
#8711	FTS: CLI status cannot parse Waiting or Hold states
#8715	FTS: calls to the service should be logged uniformly
#8783	FTS: agents EnableHold property is not read correctly
#8787	FTS: FTS alone should not transition to FinishedDirty
#8826	FTS: if setDone fails on the destination, glite-url-copy flags as successful
#8878	FTS: Occasional infinite loops in glite-url-copy process
#8884	FTS: if SRMs hang, we should timeout
#8966	FTS: srm failures are not always detected and a NULL TURL is returned
#8968	FTS: if SRM get or put retries reach the max timeout, it didn't report as failed
#8969	FTS: don't fail hard if SRM putdone fails on source
#9237	FTS: glite-transfer-list should have -a option
#9412	FTS: wrong var in default config file (watchdog config)
#9511	FTS: need to be able to specify transfer timeouts
#9622	FTS: glite-transfer-channel-list command from the UI hangs forever
#9811	FTS: getRoles method segfaults
#9825	FTS: default VETO_MAPFILE is not empty
#9948	FTS: deployment script prefixes instance names with default:
#9951	FTS: FTA wrong log file name for vo agent
#9969	Java and Perl FTS APIs not installed in WN
#9984	FTS: FPS and FTS configuration is still on a per vo instance basis
#9999	FTS: canceling a job in Waiting state corrupts that job's state
#10000	FTS/FPS: I don't like the name
#10207	FTS: rgma service discovery configuration fails
#10225	FTS: profile.d script sets X509_USER_KEY pointing to the daemon user's .globus directory
#10248	FTS: 1.3: getRoles CLI has problems
#10250	FTS: 1.3: Timestamp problem on submitTime
#10529	FTS: get a SOAP exception if using Channel endpoint to submit jobs
#10592	FTS: FTA transfers do not use delegation by default
#10625	FTS: web service installer also installs agents