gLite Data Management File Transfer Service

1. Release Description

This release contains the gLite Data Management File Transfer Service module v. 3.1.0. The following sections provide additional information about the release content, the module dependencies, the know bugs and issues and a list of bugs closed since the previous release. For information about installing and using the gLite Data Management File Transfer Service, please refer to the gLite Installation and User Guides.

2. Changes in this Release

2.1. Changes in functionality

The main changes in this release are:

Added schema upgrade 2.1.0 to 2.2.0. Fix 2.0.0->2.1.0 upgrade script.
Make the srm get and put calls simultaneous if both source and target are srms (increased performance)
Don't fail if there were connection problems, do the timeout instead (retry)
Retry the advisory delete now as well (time out)
Fix a cleanup problem in the case 'put' fails (found while reviewing the code)
Updated to match delegation interface changes
Use new VO configuration model with all VO-specific parameters grouped in a single place (vo-list.cfg.xml)

2.2. Changes in Configuration

All parameters used to set VO-specific values have been removed from the service configuration files and moved to a dedicated VO configuration file called vo-list.cfg.xml. A template is available in /opt/glite/etc/config/templates. This file is common to all gLite services running on this node or referenced in the same site-config file (unless locally overridden). For more information please refer to the gLite 1.5 installation guide.

The R-GMA Service Publisher (aka R-GMA servicetool) parameters are now automatically handled by the configuration script and do not need to be explicitly set anymore in the configuration file. The parameters can be set in the configuration file to override the default values.

3. Release contents

The gLite Data Management File Transfer Service v. 3.1.0 is composed of the following gLite components:

Component name	Description	Version	File
glite-config	gLite configuration scripts	1.6.20	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-config-1.6.20-2.noarch.rpm
glite-data-catalog-api-java	GLite Catalog Services Java client	2.0.0	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-data-catalog-api-java-2.0.0-7.noarch.rpm
glite-data-common	common files for gLite Data Management web-services	1.2.2	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-data-common-1.2.2-1.noarch.rpm
glite-data-transfer-fts	gLite File Transfer Service (FTS)	3.2.0	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-data-transfer-fts-3.2.0-6.noarch.rpm
glite-data-transfer-interface	gLite Transfer Interfaces	3.1.0	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-data-transfer-interface-3.1.0-3.noarch.rpm
glite-file-transfer-service-client-config	gLite File Transfer Service client installation	1.2.0	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-file-transfer-service-client-config-1.2.0-2.noarch.rpm
glite-rgma-servicetool-config	gLite R-GMA servicetool installation	5.2.2	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-rgma-servicetool-config-5.2.2-1.noarch.rpm
glite-security-trustmanager	The java certificate path checkin for proxy certs in SSL with plugins for tomcat and axis.	1.8.2	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-security-trustmanager-1.8.2-1.noarch.rpm
glite-security-util-java	The java utilities library for security	1.3.0	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-security-util-java-1.3.0-1.noarch.rpm
glite-security-utils-config	gLite Security Utilities configuration files	1.2.2	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-security-utils-config-1.2.2-1.noarch.rpm
glite-service-discovery-config	gLite Service Discovery installation	1.1.1	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-service-discovery-config-1.1.1-1.noarch.rpm
glite-file-transfer-service-config	gLite File Transfer Service installation	3.1.0	../../../../../../../glite-web/egee/packages/R1.5/R20051130/bin/rhel30/noarch/RPMS/glite-file-transfer-service-config-3.1.0-1.noarch.rpm

4. Dependencies

The gLite File Transfer Service module v. 3.1.0 has the following dependencies:

Component name	Description	Version	File
boost	Boost C++ libraries	1.32.0	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/boost-1.32.0-1_EGEE.noarch.rpm
gridsite	GridSite	1.1.15	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/gridsite-1.1.15-1.noarch.rpm
j2re	Java JRE	1.4.2	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/j2re-1_4_2_08-linux-i586.rpm
tomcat5	Tomcat application server	5.0.28	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/tomcat5-5.0.28-11_EGEE.noarch.rpm

The Oracle InstantClient libraries and executables are not explicitly added as dependencies in the RPMS, but they have to be installed before using the File Transfer Service with an Oracle backend. The Oracle Instantclient is not freely redistributable and is not included with this release. The required RPMS or tarballs have to be downloaded from the Oracle web site and installed on the FTS node.

5. Known bugs and issues

This release has the following known bugs and issues. Bug numbers refer to the gLite Bug Tracking system database hosted on the CERN Savannah system at https://savannah.cern.ch/bugs/?group=jra1mdw .

gLite I/O and gLite FTS security models conflict
FTS: channel agent blocks doing nothing.
Symptom: there is a procces thread only visible using ps auxm (not visible using ps aux) consuming 50% of the CPU constantly running for several hours now. Being investigated
FTS: SRM copy appends to failed logfile after failing
FTS: If SRM put call fails, but does return any requestID, we should call SRM put Done
FTS: memory leak in daemons
FTS: InfoProvider generates wrong information for channels
FTS: glite-url-copy performs an advisorydelete when the srm put fails since the target SURL already exists
FTS: FTS does not set the filesize on the SRM.put.
This stops transfers to any dCache site with strict space reservation enabled. In this case a put request with zero bytes requested really means zero bytes requested and the gridFTP is killed immeditaerly upon attempting to write.
No removal procedure is provided with this release apart from the removal of the RPMS. Any account, group or other resource created during the module configuration must be manually cleaned.

Known open bugs:

Bug number	Description
#7182	LTS deployment module does not set the log file name
#8345	gLite I/O and gLite FTS security models conflict
#8714	FTS: should be able to specify log4j file in config
#8725	When configuring the File Transfer Service (Oracle) an ambiguous result was obtained
#9165	FTS: WEB UI does not show all possible file states
#9222	FTS: Logrotate script should not print on STDOUT
#10223	In the FPS documentation and release notes there is no mention of the agent.type configuration parameter
#10304	FTS: methods for service stats do not properly process time intervals
#10812	FTS: transfer-list should return all authorized transfers
#10881	FTS: config files should not be readable by the whole world
#11216	FTS: default name for veto mapfile is wrong
#11393	FTS: The FTS should check whether a file is already being transferred
#11589	FTS: 'glite-transfer-status -l' for jobs with 100 files returns error string
#11703	FTS: installation guide does not say channels need to be created
#12389	LD_LIBRARY_PATH was not set when configuring the 1.4 FTS
#12404	Error in the 1.4 FTS documentation: the cancel map file is not to be found
#12409	1.4 FTS documentation does not clarify whether a "pool like account" should be created or not
#12503	In the documentation of the 1.4 FTS the mysql commands have a typo so that they fail when cut and pasted
#12601	1.4 FTS documentation does not describe the authentication difference between the fps and fts modes
#12627	FTS: channel agent blocks doing nothing
#12629	FTS: lock contention between VO and channel agents
#12749	FTS: 1.3 upgrade to 1.4 needs to b hacked - old endpoint cannot be maintained
#13206	FTS: SRM copy appends to failed logfile after failing
#13277	FTS: If SRM put call fails, but does return any requestID, we should call SRM put Done
#13368	FTS: reason for job status should be other than <None>
#13461	FTS: url-copy needs to provide SRM v2 support
#13479	FTS: please add a stats_generation_interval config param to config script
#13481	FTS: global restart (e.g. logrortate) should call 'restart' all, rather than 'stop' all followed by 'start all'
#13482	FTS: shouild be able to call reconfig / kill on config_service
#13483	FTS: Operating procedures would be somewhat normailized if we could pass the --instance parameter via the gLite service
#13485	FTS: should be able to disable individual agent without removing its config
#13486	FTS: watchdog script shouldn't restart forever
#13487	FTS: need to be able to visualise the current enable/disable state of agents
#13490	RGMA service discovery does not work on the FTS
#13496	FTS: when canceling an ongoing srmcopy transfer, the related log file is not moved to the failed folders
#13812	Error reports should be improved in the FTS config-migrator script
#13840	FTS: the --instance option of the deployment script does not work
#13857	FTS: global restart script doesn't produce any [OK] / [FAILED] output
#13858	FTS: can't pass individual --instance restart
#14180	FTS: The transfer aborts if the TCP buffersize parameter is set for gsiftp
#14230	FTS Config Script: start behaves like restart
#14271	FTS: InfoProvider generates wrong information for channels
#14341	FTS: WS needs log expiration/zipping
#14359	FTS: glite-url-copy performs an advisorydelete when the srm put fails since the target SURL already exists
#14491	FTS: FTS does not set the filesize on the SRM.put

Bugs fixed in this or previous releases but not yet tested:

Bug number	Description
#7503	FTS: Add TURL:TURL to DB
#8372	FTS: Security enhancements for the portal
#8379	FTS: MyProxy version is not compatible with the VDT version on LCG-2
#8588	FTS: CLI transfer-submit option -g produces Out of memory.
#8589	FTS: transfer-submit -f FILE doesn't recognize newline as delimiter
#8713	FTS: when FTS is misconfigured, the client gets obscure error messages
#8906	FTS: auto-generated quattor template has a couple of bugs
#8911	FTS: often the number of bytes transferred is incorrect in the DB
#8913	FTS: channel-set CLI should shoud allow Halted, Drain and Stopped
#8916	FTS: getVersion gives strange error
#8958	FTS: If a mapfile is not specified in the config, a client will receive an InternalException from the service
#8986	FTS: Should build C/C++ code with debug symbols and RPM should not strip the symbols from the code
#9032	FTS: primary timeout doesn't work on FTS url-copy when SRM fails on putDone
#9036	FTS: the version reported by the CLI from the webservice is incorrect
#9181	FTS: service restart does not always restart FTS agent
#9191	FTS: CLI tools need timeouts
#9201	FTS: glite-url-copy-cleanlog usage message issues
#9223	FTS: glite-url-copy print out perf markers in blocking mode?
#9224	FTS: Can't copy large files to dCache
#9232	FTS: glite-url-copy doesn't timeout if file never transitions from pending
#9240	FTS: glite-url-copy does not distinguish in FINAL message on timeouts
#9261	FTS: summary of what hapenning on a channel
#9309	FTS: dCache returns error on gridftp call, but glite-url-copy doesn't see it?
#9310	FTS: two FINAL messages if advisoryDelete fails against dCache
#9322	FTS: Formatting could be better on new log messages
#9338	FTS: agent shoud fill in DN field when it updates a table
#9398	FTS: mysterious failure on PUT when it apparently completed OK
#9399	FTS: blocking glite-url-copy segfaults or buffer overrruns sometimes on srm-putdone error
#9480	FTS: running changeStateForHeldJobs("Failed") corrupts jobstate
#9485	FTS: job/file state corruption in DB
#9500	FTS: FTS daemon lockfile prevents a restart from watchdog script
#9503	FTS: 1.3: gsiftp - gsiftp transfer fails
#9531	FTS: 'reason' buffer is only 1024 bytes, and if overfilled results in Oracle error on update
#9549	FTS: actually use the numbrer of streams per channel setting
#9712	FTS: when myproxy retrieval fails, the DB connection sometimes drops
#9732	FTS: should periodically refresh its mapfiles
#9789	FTS: submit command always prompts for the myproxy password
#9793	FTS: FTA wrong default value for vo-agent
#9812	FTS: perl APIs should work
#9813	FTS: 1.3: url-copy: SRM get failure seg faults.
#9821	FTS: Client code libraries should not expose FilePlacement or TransferFetcher interface
#9826	FTS: org.glite.data: Error while reading directory null
#9828	FTS: services.xml with wrong XML syntax makes glite-transfer commands to hang forever
#9833	FTS: It'd be clearer for deployment to have FTS hostcert in /etc/grid-security/
#9835	FTS: VETO mapfile entries allow channel-add but not channel-list
#9839	FTS: should probably switch VETOed people to include Admins
#9968	FTS: the deployment script does not detect if an incompatible schema version is present
#10060	FTS: FTA 1.3: does not retry submission after the retry interval time
#10076	FTS: should return error immediately if host is not SRM
#10119	FTS: Full table scan on DB is limiting the throughput rate
#10198	FTS: Should not be possible to open more than one channel between 2 sites
#10222	FTS: DB connection should be per instance
#10235	FTS: must be able to stop individual instances without stopping the others
#10249	FTS: daemon sometimes aborts when password file is being rebuilt by Quattor
#10329	FTS: FTA doesn't retry transfers when urlcopy doesn't return a reason class
#10362	cannot submit fts jobs using delegated proxies
#10440	FTS: ChannelAgent doesn't always use client delegated credentials for srmcopy
#10479	FTS: cleanlog script does completed when asked to do failed
#10490	FTS: FTA channel agent with MySQL backend don't start
#10510	FTS: glite-transfer-list returns the same job id status multiple times
#10534	FTS: FTA MySQL: agents get the wrong type
#10557	FTS: transfers with SRM copy fail
#10633	FTS: myproxy should be an "associated service" for a FTS
#10721	FTS: duration not in interface
#10819	FTS: not all web-service methods use SELECT FOR UPDATE locking
#10823	FTS: FPS doesn't contact the catalog with the user delegated credentials and then dies
#10834	FTS: copy client doesn not handle problems during error recovery
#10848	FTS: Submission using non-voms proxies not working
#10909	FTS: should publish channel-keyed information into BDII
#10912	FTS: check for request manager/owner not done properly (MySQL)
#10914	FTS: configuration scripts should not override existing service discovery file
#10953	FTS: glite-transfer-stats-agents segfaults if last_active is missing
#11021	FTS: add SRM request Id in case of error
#11065	FTS: Sometimes t_transfer reports a file as Active while the job (and file) are in Hold
#11087	FTS: fts-gridmap.sh erases the mapfile if glite-mkgridmap error
#11188	FTS: INSTALL file still contains reference to old endpoint
#11221	FTS: inefficient query on agent can stop transfers
#11472	FTS: transfer-cli man pages do not build - missing xml files
#12469	FTS: sometimes the client gets an error that says 'success'
#12582	FTS: info-provider script doesn't provide a vital attribute
#12602	FTS: 1.4 FTA does not work from castor to castor with srmcopy
#12747	FTS CLI: glite-transfer-stats-* should use correct month formats
#12961	FTS: FTA uses the wrong service type for GridFTP services
#13301	FTS: FTA doesn't respect the transfer limit imposed in the channel table
#13310	FTS: memory leak in daemons
#13380	FTS Info provider does not get channel access info from DB
#13724	FTS: Inconsistent jobs' state in FTA
#13814	FTS: 1.4.1 schema does not install
#13838	FTS: typos in agent channel log file
#13841	FTS: -h option does not say which are the L commands

6. Bugs closed since last release

This release fixes the following bugs and issues. Bug numbers refer to the gLite Bug Tracking system database hosted on the CERN Savannah system at https://savannah.cern.ch/bugs/?group=jra1mdw

Bug number	Description
#7354	FTS: glite_config_service waits for all child transfer processes to finish before stopping
#7502	FTS: Channel management HALTED state should be set on repeated failures
#7516	FTS: log rotation on service
#8333	FTS: service is not able to parse correctly the "grid-mapfile" with the administrators of the service
#8364	FTS: Accept not fully qualified SURLs on the server side
#8367	FTS: Handling of full stage pools not optimal
#8368	FTS: Improve retry and channel halting mechanisms
#8369	FTS: Validation of FTS for DB deployment
#8371	FTS: Usage of MySQL
#8374	FTS: Usage of SRM Copy
#8377	FTS: Trace facility
#8378	FTS: Cleanup of jobs in terminal states in the database
#8380	FTS: Interaction with network monitoring
#8381	FTS: Alarm script to notice mysterious drop in throughput
#8383	FTS: Knowledge of the service portal needed
#8385	FTS: Multi hop means no async transfers
#8587	FTS: (and others) Oracle connections can go stale
#8795	FTS: any glite-transfer-channel-set command sets the channel Active
#8915	FTS: fts starting script reports success when it fails
#8947	FTS: Throughput phase: need a post-transfer agent to remove files from disk
#8949	FTS: max number of tomcat DB connections in pool should match DB deployment maximum
#9121	FTS client installation
#9166	FTS: No simple way to list all requestID's in a state
#9200	FTS: glite-url-copy uses both jar and tar
#9203	FTS: Would be useful to have a view of ongoing transfers
#9207	FTS: glite-url-copy arguments are order sensitive
#9236	FTS: glite-url-copy segfaults on readdir if directory does not exist.
#9241	FTS: Put request ID from SRM into log
#9499	FTS - Client ignores invalid host certificate.
#9625	FTS: fails if double slash in filename
#9787	FTS: submitting a transfer wihout a proxy gives a SOAP exception
#9834	FTS: good to have backwards compatible interface ports on FTS
#9883	FPS: the interface has to be able to accept FTS jobs.
#9913	Oracle instantclient RPM deps should be removed from all modules
#9938	FTS: BDII generic information needed (SD is not sufficient).
#9939	FTS: Need to finish BDII integration
#9943	FTS: clients report "Missing SD plugin" if any plugin fails
#9950	FTS: FTS deployment script --status option does not work
#9952	FTS: submit client should ask for MyProxy password twice
#9959	FTS: scripts to check SRM timings, etc
#9967	FTS: drop Schema for old versions missing in rpm
#10191	FTS: would be good to get rid of count in DB
#10192	FTS: would be nice to have state triggers
#10200	FTS: dropping of a channel with jobs should not give internal server error
#10201	FTS: glite-transfer-channel-add usage needs to be updated
#10209	FTS: client should catch SOAP exception if service is down
#10221	FTS: schema load file should not drop the existing schema
#10236	FTS: getroles CLI segfaults
#10279	FTS: glite-transfer-submit produce out of memory if the job description file is empty
#10280	FTS: glite-transfer-submit does not detect absence of destination SURL
#10320	FTS: client specifies the number of retries
#10436	FTS: errors in FTS CLI not signalled as such
#10569	FTS: MySQL FTS Service error: VO 'egtest' agent not running.
#10582	FTS: FTA SRM copy log file are way too verbose...
#10588	FTS: FTA canceled job does not want to die...
#10634	FTS: FTS must publish which channels it serves
#10720	FTS: FTA. SRM copy: not all requests are filled with duration
#10727	FTS: VOMS role based authz does not work
#10769	FTS: should allow user selection of which MYPROXY_SERVER to use
#10855	FTS: srm copy sometimes hangs on srm copy Pending
#10910	FTS: server (agents) should use information system to find SRM information
#10936	FTS: SC4 expects SRMs to run SRM v2 - FTS should support it on these timescales
#11217	FTS: Why can't I do X? Review authorisation permissions on web-service
#12219	./glite-file-transfer-service-config.py --configure yields an error when configuring mysql FTS
#13859	FTS: /opt/glite/etc/config/scripts/glite-file-transfer-agents-config.py --help doesn't list 'add' as one of the options
#13860	FTS: GSIFTP - GSIFTP transfers don't work and put failure is ignored