gLite Data Management File Transfer Service (Oracle)

1. Release Description

This release contains the gLite Data Management File Transfer Service for Oracle module v. 2.0.2. The following sections provide additional information about the release content, the module dependencies, the know bugs and issues and a list of bugs closed since the previous release. For information about installing and using the gLite Data Management File Transfer Service, please refer to the gLite Installation and User Guides.

2. Changes in this Release

The major change of the File Transfer Service is that it does not contain the transfer agents anymore. A separate module (Transfer Agents Oracle) has been released for the agents. Apart from that the main changes in this release are:

A watchdog mechanism has been added to monitor the running services and restart them in cas of problems. The watchdog is enabled by using the enable.watchdog parameter and it’s configured by specifying the watchdog.fts.check-command and watchdog.fts.return-string parameters in each configured instance
The –configure option has been added. Once the configuration script is executed with the –configure option, in order to start the services it is necessary to run the configuration script with the –start option.
It has been added the –status option to show the services.
Bug fixes and improvements to the service.

3. Release contents

The gLite Data Management File Transfer Service v. 2.0.2 is composed of the following gLite components:

Component name	Description	Version	File
glite-config	gLite configuration scripts	1.2.1	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/noarch/RPMS/glite-config-1.2.1-1.noarch.rpm
org.glite.data-catalog-api.c	gLite Data Catalogs C/C++ API	2.0.0	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/i386/RPMS/glite-data-catalog-api-c-2.0.0-2.i386.rpm
org.glite.data-catalog-api.java	GLite Catalog Services Java client	2.0.0	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/noarch/RPMS/glite-data-catalog-api-java-2.0.0-6.noarch.rpm
org.glite.data-catalog.cli	Data Catalog CLI	1.7.0	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/i386/RPMS/glite-data-catalog-cli-1.7.0-2.i386.rpm
org.glite.data-catalog.interface	gLite Catalog Interfaces	2.0.0	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/noarch/RPMS/glite-data-catalog-interface-2.0.0-6.noarch.rpm
org.glite.data-common	common files for gLite Data Management web.services	1.2.1	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/noarch/RPMS/glite-data-common-1.2.1-1.noarch.rpm
org.glite.data-config.service	GLite Data Config Service	1.1.8	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/i386/RPMS/glite-data-config-service-1.1.8-1.i386.rpm
org.glite.data-srm-api.c	gLite Data SRM C/C++ API	1.0.0	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/i386/RPMS/glite-data-srm-api-c-1.0.0-15.i386.rpm
org.glite.data-transfer.agents	gLite Data Transfer Agents	1.9.2	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/i386/RPMS/glite-data-transfer-agents-1.9.2-3.i386.rpm
org.glite.data-transfer-api.c	GLite Data Transfer C/C++ API	2.9.0	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/i386/RPMS/glite-data-transfer-api-c-2.9.0-2.i386.rpm
org.glite.data-transfer.cli	Data Transfer CLI	1.3.8	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/i386/RPMS/glite-data-transfer-cli-1.3.8-2.i386.rpm
org.glite.data-transfer.common	gLite Transfer Services Common library	2.3.1	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/noarch/RPMS/glite-data-transfer-common-2.3.1-1.noarch.rpm
org.glite.data-transfer.fts	gLite File Transfer Service (FTS)	2.3.1	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/noarch/RPMS/glite-data-transfer-fts-2.3.1-2.noarch.rpm
org.glite.data-transfer.interface	gLite Transfer Interfaces	2.9.0	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/noarch/RPMS/glite-data-transfer-interface-2.9.0-1.noarch.rpm
org.glite.data-transfer-url.copy	gLite Data Transfer URL Copy	1.2.6	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/i386/RPMS/glite-data-transfer-url-copy-1.2.6-1.i386.rpm
org.glite.data-util.c	Data C util libraries	1.1.1	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/i386/RPMS/glite-data-util-c-1.1.1-1.i386.rpm
org.glite.file-transfer-service-oracle.config	@build.package.summary@	2.0.2	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/noarch/RPMS/glite-file-transfer-service-oracle-config-2.0.2-1.noarch.rpm
org.glite.rgma-api-java	Java API for R.GMA	4.1.5	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/noarch/RPMS/glite-rgma-api-java-4.1.5-1.noarch.rpm
org.glite.rgma-base	R.GMA basic configuration and documentation	4.1.19	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/noarch/RPMS/glite-rgma-base-4.1.19-1.noarch.rpm
org.glite.rgma-common.config	gLite rgma common configuration items installation	5.0.0	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/noarch/RPMS/glite-rgma-common-config-5.0.0-1.noarch.rpm
org.glite.rgma-servicetool	R.GMA service tool	4.1.19	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/noarch/RPMS/glite-rgma-servicetool-4.1.19-3.noarch.rpm
org.glite.rgma-servicetool-config	gLite R.GMA servicetool installation	5.0.0	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/noarch/RPMS/glite-rgma-servicetool-config-5.0.0-1.noarch.rpm
org.glite.rgma-stubs-servlet-java	Java client implementation stubs for R.GMA	4.1.12	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/noarch/RPMS/glite-rgma-stubs-servlet-java-4.1.12-1.noarch.rpm
org.glite.security.trustmanager	The java certificate path checkin for proxy certs in SSL with plugins for tomcat and axis.	1.7.3	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/noarch/RPMS/glite-security-trustmanager-1.7.3-1.noarch.rpm
org.glite.security-util.java	The java utilities library for security	1.1.2	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/noarch/RPMS/glite-security-util-java-1.1.2-2.noarch.rpm
org.glite.security-utils.config	gLite Security Utilities configuration files	1.0.2	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/noarch/RPMS/glite-security-utils-config-1.0.2-2.noarch.rpm
org.glite.security-voms-api.c		1.5.9	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/i386/RPMS/glite-security-voms-api-c-1.5.9-2.i386.rpm
org.glite.service-discovery-api.c	C API for Service Discovery.	2.0.0	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/i386/RPMS/glite-service-discovery-api-c-2.0.0-0.i386.rpm
org.glite.service-discovery-file.c	File based Service Discovery	2.0.0	../../../../../../../glite-web/egee/packages/R1.2/R20050715/bin/rhel30/i386/RPMS/glite-service-discovery-file-c-2.0.0-1.i386.rpm

4. Dependencies

The gLite File Transfer Service module v. 2.0.2 has the following dependencies:

Component name	Version	RPM file name
gLite C/C++ Essentials	1.1.1	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/glite-essentials-cpp-1.1.1-1_EGEE.i386.rpm
GPT	VDT 1.2.2	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/gpt-VDT1.2.2rh9-1.i386.rpm
VDT Globus Essentials	VDT 1.2.2	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/vdt_globus_essentials-VDT1.2.2rh9-1.i386.rpm
CGSI_gSOAP_2.6	1.1.8	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/CGSI_gSOAP_2.6-1.1.8-2.slc3.i386.rpm
perl-Expect.pm	1.01	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/perl-Expect.pm-1.01-9.i386.rpm
myproxy	1.14	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/myproxy-1.14-EGEE.i386.rpm
Tomcat	5.0.28	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/tomcat5-5.0.28-9_EGEE.i386.rpm
Java JRE/SDK	1.4.2	http://java.sun.com/j2se/1.4.2/download.html
Oracle InstantClient	10.1.0.4	http://www.oracle.com/technology/software/tech/oci/instantclient/htdocs/linuxsoft.html

The Oracle InstantClient libraries and executables are not freely redistributable and are not included with this release. The four RPMS have to be downloaded from the Oracle web site and installed on the system before installing the gLite File Transfer Service module.

5. Known bugs and issues

This release has the following known bugs and issues. Bug numbers refer to the gLite Bug Tracking system database hosted on the CERN Savannah system at https://savannah.cern.ch/bugs/?group=jra1mdw .

The gLite I/O and gLite FTS security models conflict. This is being discussed in several coordination bodies. A proposal to harmonize them has been already produced and circulated as part of the updated gLite Architecture document
FTS has no inter-VO scheduling
For service discovery to work for FTS clients, the File Transfer Service instances must be published (in R-GMA or files) using the service type org.glite.FileTransfer and org.glite.ChannelManagement, instead of the types org.glite.data.FileTransfer org.glite.data.ChannelManagement mandated by the gLite service publishing conventions. This will be fixed in a future release. The provided configuration templates use org.glite.FileTransfer and org.glite.ChannelManagement for the moment
No removal procedure is provided with this release apart from the removal of the RPMS. Any account, group or other resource created during the module configuration must be manually cleaned.

Known open bugs:

Bug number	Description
#6935	Install guide mentions RGMA-Server in LTS Section
#7182	LTS deployment module does not set the log file name
#7353	FTS: if XML config file is badly formatted, the glite_config_service tool hangs on start
#7502	FTS: Channel management HALTED state should be added
#7503	FTS: Add TURL:TURL to DB
#7516	FTS: log rotation on service
#8345	gLite I/O and gLite FTS security models conflict
#8364	FTS: Accept not fully qualified SURLs on the server side
#8366	FTS: Transfer Agent for a channel is per-VO
#8367	FTS: Handling of full stage pools not optimal
#8368	FTS: Improve retry and channel halting mechanisms
#8369	FTS: Validation of FTS for DB debployment
#8371	FTS: Usage of MySQL
#8372	FTS: Security enhancements for the portal
#8374	FTS: Usage of SRM Copy
#8377	FTS: Trace facility
#8378	FTS: Cleanup of jobs in terminal states in the database
#8379	FTS: MyProxy version is not compatible with the VDT version on LCG-2
#8380	FTS: Interaction with network monitoring
#8381	FTS: Service Level monitoring
#8382	FTS: Split deployment model
#8383	FTS: Knowledge of the service portal needed
#8384	FTS: Not clear if single file to multiple destinations is done well
#8385	FTS: Multi hop means no async transfers
#8386	FTS: No inter-VO scheduling
#8587	FTS: (and others) Oracle connections can go stale
#8714	FTS: should be able to specify log4j file in config
#8725	When configuring the File Transfer Service (Oracle) an ambiguous result was obtained
#8906	FTS: auto-generated quattor template has a couple of bugs
#8947	FTS: Throughput phase: need a post-transfer agent to remove files from disk
#8949	FTS: max number of tomcat DB connections in pool should match DB deployment maximum
#8958	FTS: If a mapfile is not specified in the config, a client will receive an InternalException from the service
#8986	FTS: Should build C/C++ code with debug symbols and RPM should not strip the symbols from the code
#9036	FTS: the version reported by the CLI from the webservice is incorrect
#9203	FTS: Would be useful to have a view of ongoing transfers
#9222	FTS: Logrotate script should not print on STDOUT
#9259	FTS: allow channel level canceling
#9261	FTS: summary of what hapenning on a channel
#9338	FTS: agent shoud fill in DN field when it updates a table
#9407	FTS: logrotate has strange errors
#9411	FTS: glite-file-transfer-service-oracle-config.py crash if directory is missing
#9480	FTS: running changeStateForHeldJobs("Failed") corrupts jobstate
#9485	FTS: job/file state corruption in DB
#9499	FTS - Client ignores invalid host certificate.
#9500	FTS: FTS daemon lockfile prevents a restart from watchdog script
#9503	FTS: gsiftp - gsiftp transfer fails
#9531	FTS: 'reason' buffer is only 1024 bytes, and if overfilled results in Oracle error on update
#9549	FTS: actually use the numbrer of streams per channel setting
#9552	FTS: glite-url-copy dies when > 10 streams to dCache targets
#9622	FTS: glite-transfer-channel-list command from the UI hangs forever
#9625	FTS: fails if double slash in filename

Bugs fixed in this or previous releases but not yet tested:

Bug number	Description
#6931	FTS: Should use user proxy for transfer
#6932	FTS: Should support Oracle for its back end database
#6933	FTS: Should support Channel management
#7113	FTS: glite-url-copy - if SRM.put fails, the code loops forever
#7318	FTS: Transfer agent sometimes mis-counts the number of active transfers
#7352	FTS: forked glite-url-copy should chdir somewhere it can write
#7354	FTS: glite_config_service waits for all child transfer processes to finish before stopping
#7355	FTS: transfer-url-copy sometimes dies on gridFTP
#7358	FTS: channel allocation fails if the SURL doesn't contain a port number
#7361	FTS: transfer agent should detect a bad Oracle connection, and try to reconnect
#7466	FTS: transfer-url-copy returns 0 for size and bytes written
#7506	FTS: requirement 1.2 should be met: proper retry
#7508	FTS: now needs to implement the protocol option passing as defined in interface
#7517	FTS: rename FilePlacement ot FileTransfer, fps -> fts
#7568	FTS: url-copy sometimes core dumps if SURL format is bad
#7581	FTS: add support for error categories
#7594	FTS: should migrate all components to FileTransfer interface
#7595	FTS: Need to test installation of -fts module
#7596	FTS: Commandline tools should be tested fully against service
#7738	FTS: clean up SRM after transfer failure
#7739	FTS: time out on Gridftp if it hangs on network failure
#7740	FTS: glite-url-copy dumps core on some SRM failures
#8333	FTS: service is not able to parse correctly the "grid-mapfile" with the administrators of the service
#8336	FTS: testing methods do not work
#8365	FTS: Command Lines need service resolution from BDII
#8373	FTS: Certification with dCache and DPM
#8375	FTS: Daemon watchdog scripts
#8376	FTS: per transfer logs need better management
#8486	FTS: submit commandline doesn't accept myproxy password
#8503	FTS: detailed transfer-status commandline doesn't work correctly
#8504	FTS: failure message for channel-list could be better
#8538	FTS: channel-set -S Active does not work.
#8588	FTS: CLI transfer-submit option -g produces Out of memory.

#8589	FTS: transfer-submit -f FILE doesn't recognize newline as delimiter
#8592	FTS: command-line and java service mispelling of Canceling and Canceled
#8633	the FTS configuration script does not run the rgma script
#8670	FTS: SC3 Quattor template has missing deps
#8690	FTS: channel management setStatusForHeldJobs has problems
#8701	FTS: server should accept equalsIgnoreCase("gridftp") for the transfer params
#8702	FTS: default portal logging should be DEBUG
#8707	FTS: Deployment script should be update in light of discussions with DB people
#8708	FTS: transfer-cli tools should not depend on catalog-cli tools
#8711	FTS: CLI status cannot parse Waiting or Hold states
#8713	FTS: when FTS is misconfigured, the client gets obscure error messages
#8715	FTS: calls to the service should be logged uniformly
#8783	FTS: agents EnableHold property is not read correctly
#8787	FTS: FTS alone should not transition to FinishedDirty
#8795	FTS: any glite-transfer-channel-set command sets the channel Active
#8826	FTS: if setDone fails on the destination, glite-url-copy flags as successful
#8878	FTS: Occassional infinite loops in glite-url-copy process
#8884	FTS: if SRMs hang, we should timeout
#8911	FTS: often the number of bytes transferred is incorrect in the DB
#8913	FTS: channel-set CLI should shoud allow Halted, Drain and Stopped
#8915	fts reports success when it fails
#8966	FTS: srm failures are not always detected and a NULL TURL is returned
#8968	FTS: if SRM get or put retries reach the max timeout, it didn't report as failed
#8969	FTS: don't fail hard if SRM putdone fails on source
#9032	FTS: primary timeout doesn't work on FTS url-copy when SRM fails on putDone
#9166	FTS: No simple way to list all requestID's in a state
#9207	FTS: glite-url-copy arguments are order sensitive
#9223	FTS: glite-url-copy print out perf markers in blocking mode?
#9224	FTS: Can't copy large files to dCache
#9232	FTS: glite-url-copy doesn't timeout if file never transitions from pending
#9236	FTS: glite-url-copy segfaults on readdir if directory does not exist.
#9237	FTS: glite-transfer-list should have -a option
#9240	FTS: glite-url-copy does not distinguish in FINAL message on timeouts
#9241	FTS: Put request ID from SRM into log
#9309	FTS: dCache returns error on gridftp call, but glite-url-copy doesn't see it?
#9310	FTS: two FINAL messages if advisoryDelete fails against dCache
#9322	FTS: Formatting could be better on new log messages
#9398	FTS: mysterious failure on PUT when it apparently completed OK
#9399	FTS: blocking glite-url-copy segfaults or buffer overrruns sometimes on srm-putdone error
#9412	FTS: wrong var in default config file (watchdog config)
#9511	FTS: need to be able to specify transfer timeouts
#9521	FTS: remove RPM deps on Oracle instantclient

6. Bugs closed since last release

This release fixes the following bugs and issues. Bug numbers refer to the gLite Bug Tracking system database hosted on the CERN Savannah system at https://savannah.cern.ch/bugs/?group=jra1mdw

Bug number	Description
#7509	FTS: update security per Channel
#8370	FTS: Application Server deployment
#8722	When configuring the File Transfer Service (Oracle) service the enviornment script is not sourced
#8724	After the File Transfer Service (Oracle) installation the script gLite is found to be in /opt/glite/etc/conf/script
#8876	FTS: null or empty soapfault detail from SRM segfaults glite-url-copy process
#9121	FTS client installation