gLite Data Management File Transfer Service (Oracle)

1. Release Description

This release contains the gLite Data Management File Transfer Service for Oracle module v. 2.1.0. The following sections provide additional information about the release content, the module dependencies, the know bugs and issues and a list of bugs closed since the previous release. For information about installing and using the gLite Data Management File Transfer Service, please refer to the gLite Installation and User Guides.

2. Changes in this Release

The main changes in this release are:

Cancel operation is now bulk - receives list of request ids
New functionality to update job priority within the VO (to be done by ADMIN or VO manager)
Updated configuration options:

dropped cancel/query_admin_mapfile and cancel/query_admin_voms_attribute
renamed manager_admin/submit_admin_mapfile and manager_admin/submit_admin_voms_attribute to admin/submit_mapfile and admin/submit_voms_attributes
added veto_mapfile
instead of a single voms attribute now lists of voms attributes are supported

New authorization scheme. Summary:

ADMIN - all privileges on service (including submit jobs)
Channel manager - may update channel info, query all information related to channel (including requests)
VO manager - may query all information related to VO (including requests)
Submitter - may submit jobs
Vetoed - all privileges revoked (even if client has one of the other roles above)

Channel managers and admin can set vo share on channels
New VO management functionality (add/remove/list VO managers)
New channel management functionality (add/remove/list channel managers)
Removed Oracle instant client dependency
Bug fixes and improvements to the service.

3. Release contents

The gLite Data Management File Transfer Service v. 2.1.0 is composed of the following gLite components:

Component name	Description	Version	File
glite-config	gLite configuration scripts	1.2.1	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/noarch/RPMS/glite-config-1.2.1-1.noarch.rpm
org.glite.data-catalog-api.c	gLite Data Catalogs C/C++ API	2.0.0	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/i386/RPMS/glite-data-catalog-api-c-2.0.0-3.i386.rpm
org.glite.data-catalog-api.java	GLite Catalog Services Java client	2.0.0	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/noarch/RPMS/glite-data-catalog-api-java-2.0.0-6.noarch.rpm
org.glite.data-catalog.cli	Data Catalog CLI	1.7.0	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/i386/RPMS/glite-data-catalog-cli-1.7.0-2.i386.rpm
org.glite.data-catalog.interface	gLite Catalog Interfaces	2.0.0	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/noarch/RPMS/glite-data-catalog-interface-2.0.0-6.noarch.rpm
org.glite.data-common	common files for gLite Data Management web.services	1.2.1	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/noarch/RPMS/glite-data-common-1.2.1-1.noarch.rpm
org.glite.data-config.service	GLite Data Config Service	1.1.9	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/i386/RPMS/glite-data-config-service-1.1.9-1.i386.rpm
org.glite.data-srm-api.c	gLite Data SRM C/C++ API	1.1.0	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/i386/RPMS/glite-data-srm-api-c-1.1.0-1.i386.rpm
org.glite.data-transfer.agents	gLite Data Transfer Agents	2.0.0	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/i386/RPMS/glite-data-transfer-agents-2.0.0-4.i386.rpm
org.glite.data-transfer-api.c	GLite Data Transfer C/C++ API	3.0.0	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/i386/RPMS/glite-data-transfer-api-c-3.0.0-2.i386.rpm
org.glite.data-transfer.cli	Data Transfer CLI	3.0.0	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/i386/RPMS/glite-data-transfer-cli-3.0.0-1.i386.rpm
org.glite.data-transfer.common	gLite Transfer Services Common library	3.0.0	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/noarch/RPMS/glite-data-transfer-common-3.0.0-5.noarch.rpm
org.glite.data-transfer.fts	gLite File Transfer Service (FTS)	3.0.0	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/noarch/RPMS/glite-data-transfer-fts-3.0.0-5.noarch.rpm
org.glite.data-transfer.interface	gLite Transfer Interfaces	3.0.0	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/noarch/RPMS/glite-data-transfer-interface-3.0.0-3.noarch.rpm
org.glite.data-transfer-url.copy	gLite Data Transfer URL Copy	1.3.0	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/i386/RPMS/glite-data-transfer-url-copy-1.3.0-1.i386.rpm
org.glite.data-util.c	Data C util libraries	1.1.2	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/i386/RPMS/glite-data-util-c-1.1.2-1.i386.rpm
org.glite.file-transfer-service-oracle.config	@build.package.summary@	2.1.0	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/noarch/RPMS/glite-file-transfer-service-oracle-config-2.1.0-1.noarch.rpm
org.glite.rgma-api-java	Java API for R.GMA	4.1.5	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/noarch/RPMS/glite-rgma-api-java-4.1.5-1.noarch.rpm
org.glite.rgma-base	R.GMA basic configuration and documentation	4.1.19	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/noarch/RPMS/glite-rgma-base-4.1.19-1.noarch.rpm
org.glite.rgma-common.config	gLite rgma common configuration items installation	5.0.0	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/noarch/RPMS/glite-rgma-common-config-5.0.0-1.noarch.rpm
org.glite.rgma-servicetool	R.GMA service tool	4.1.19	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/noarch/RPMS/glite-rgma-servicetool-4.1.19-3.noarch.rpm
org.glite.rgma-servicetool-config	gLite R.GMA servicetool installation	5.0.0	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/noarch/RPMS/glite-rgma-servicetool-config-5.0.0-1.noarch.rpm
org.glite.rgma-stubs-servlet-java	Java client implementation stubs for R.GMA	4.1.12	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/noarch/RPMS/glite-rgma-stubs-servlet-java-4.1.12-1.noarch.rpm
org.glite.security.trustmanager	The java certificate path checkin for proxy certs in SSL with plugins for tomcat and axis.	1.7.3	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/noarch/RPMS/glite-security-trustmanager-1.7.3-1.noarch.rpm
org.glite.security-util.java	The java utilities library for security	1.1.2	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/noarch/RPMS/glite-security-util-java-1.1.2-2.noarch.rpm
org.glite.security-utils.config	gLite Security Utilities configuration files	1.0.3	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/noarch/RPMS/glite-security-utils-config-1.0.3-1.noarch.rpm
org.glite.security-voms-api.c		1.5.9	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/i386/RPMS/glite-security-voms-api-c-1.5.9-2.i386.rpm
org.glite.service-discovery-api.c	C API for Service Discovery.	2.0.1	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/i386/RPMS/glite-service-discovery-api-c-2.0.1-0.i386.rpm
org.glite.service-discovery-file.c	File based Service Discovery	2.0.0	../../../../../../../glite-web/egee/packages/R1.3/R20050805/bin/rhel30/i386/RPMS/glite-service-discovery-file-c-2.0.0-1.i386.rpm

4. Dependencies

The gLite File Transfer Service module v. 2.1.0 has the following dependencies:

Component name	Version	RPM file name
gLite C/C++ Essentials	1.1.1	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/glite-essentials-cpp-1.1.1-1_EGEE.i386.rpm
GPT	VDT 1.2.2	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/gpt-VDT1.2.2rh9-1.i386.rpm
VDT Globus Essentials	VDT 1.2.2	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/vdt_globus_essentials-VDT1.2.2rh9-1.i386.rpm
CGSI_gSOAP_2.6	1.1.8	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/CGSI_gSOAP_2.6-1.1.8-2.slc3.i386.rpm
perl-Expect.pm	1.01	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/perl-Expect.pm-1.01-9.i386.rpm
myproxy	1.14	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/myproxy-1.14-EGEE.i386.rpm
Tomcat	5.0.28	../../../../../../../glite-web/egee/packages/externals/bin/rhel30/RPMS/tomcat5-5.0.28-9_EGEE.i386.rpm
Java JRE/SDK	1.4.2	http://java.sun.com/j2se/1.4.2/download.html
Oracle InstantClient	10.1.0.4	http://www.oracle.com/technology/software/tech/oci/instantclient/htdocs/linuxsoft.html

The Oracle InstantClient libraries and executables are not freely redistributable and are not included with this release. The four RPMS have to be downloaded from the Oracle web site and installed on the system before installing the gLite File Transfer Service module.

5. Known bugs and issues

This release has the following known bugs and issues. Bug numbers refer to the gLite Bug Tracking system database hosted on the CERN Savannah system at https://savannah.cern.ch/bugs/?group=jra1mdw .

The gLite I/O and gLite FTS security models conflict. This is being discussed in several coordination bodies. A proposal to harmonize them has been already produced and circulated as part of the updated gLite Architecture document
For service discovery to work for FTS clients, the File Transfer Service instances must be published (in R-GMA or files) using the service type org.glite.FileTransfer and org.glite.ChannelManagement, instead of the types org.glite.data.FileTransfer org.glite.data.ChannelManagement mandated by the gLite service publishing conventions. This will be fixed in a future release. The provided configuration templates use org.glite.FileTransfer and org.glite.ChannelManagement for the moment
No removal procedure is provided with this release apart from the removal of the RPMS. Any account, group or other resource created during the module configuration must be manually cleaned.

Known open bugs:

Bug number	Description
#7318	FTS: Transfer agent sometimes mis-counts the number of active transfers
#7353	FTS: if XML config file is badly formatted, the glite_config_service tool hangs on start
#7355	FTS: transfer-url-copy sometimes dies on gridFTP
#7466	FTS: transfer-url-copy returns 0 for size and bytes written
#7502	FTS: Channel management HALTED state should be set on repeated failures
#7503	FTS: Add TURL:TURL to DB
#7738	FTS: clean up SRM after transfer failure: advisory delete
#7739	FTS: time out on Gridftp if it hangs on network failure
#7740	FTS: glite-url-copy dumps core on some SRM failures
#8345	gLite I/O and gLite FTS security models conflict
#8364	FTS: Accept not fully qualified SURLs on the server side
#8365	FTS: Command Lines need service resolution from BDII
#8367	FTS: Handling of full stage pools not optimal
#8368	FTS: Improve retry and channel halting mechanisms
#8369	FTS: Validation of FTS for DB debployment
#8371	FTS: Usage of MySQL
#8372	FTS: Security enhancements for the portal
#8373	FTS: Certification with dCache, DPM and new castor SRM
#8374	FTS: Usage of SRM Copy
#8377	FTS: Trace facility
#8378	FTS: Cleanup of jobs in terminal states in the database
#8379	FTS: MyProxy version is not compatible with the VDT version on LCG-2
#8380	FTS: Interaction with network monitoring
#8381	FTS: Service Level monitoring
#8383	FTS: Knowledge of the service portal needed
#8385	FTS: Multi hop means no async transfers
#9165	FTS: WEB UI does not show all possible file states
#8587	FTS: (and others) Oracle connections can go stale
#8708	FTS: transfer-cli tools should not depend on catalog-cli tools
#8714	FTS: should be able to specify log4j file in config
#8947	FTS: Throughput phase: need a post-transfer agent to remove files from disk
#8949	FTS: max number of tomcat DB connections in pool should match DB deployment maximum
#8958	FTS: If a mapfile is not specified in the config, a client will receive an InternalException from the service
#9121	FTS client installation
#9203	FTS: Would be useful to have a view of ongoing transfers
#9261	FTS: summary of what hapenning on a channel
#9338	FTS: agent shoud fill in DN field when it updates a table
#9480	FTS: running changeStateForHeldJobs("Failed") corrupts jobstate
#9485	FTS: job/file state corruption in DB
#9500	FTS: FTS daemon lockfile prevents a restart from watchdog script
#9503	FTS: gsiftp - gsiftp transfer fails
#9531	FTS: 'reason' buffer is only 1024 bytes, and if overfilled results in Oracle error on update
#9549	FTS: actually use the numbrer of streams per channel setting
#9622	FTS: glite-transfer-channel-list command from the UI hangs forever
#9712	FTS: when myproxy retrieval fails, the DB connection sometimes drops
#9732	FTS: should periodically refresh its mapfiles
#9787	FTS: submitting a transfer wihout a proxy gives a SOAP exception
#9789	FTS: submit command always prompts for the myproxy password
#9811	FTS: getRoles method segfaults
#9812	FTS: perl APIs should work
#9813	FTS: 1.3.0 url-copy: SRM get failure seg faults.
#9821	FTS: Client code libraries should not expose FilePlacement or TransferFetcher interface
#9826	FTS: org.glite.data: Error while reading directory null
#9828	FTS: services.xml with wrong XML syntax makes glite-transfer commands to hang forever
#9834	FTS: good to have backwards compatible interface ports on FTS
#9835	FTS: VETO mapfile entries allow channel-add but not channel-list
#9839	FTS: should probably switch VETOed people to include Admins
#9883	FPS: the interface has to be able to accept FTS jobs.
#9938	FTS: BDII generic information needed (SD is not sufficient).
#9939	FTS: Need to finish BDII integration
#9943	FTS: clients report "Missing SD plugin" if any plugin fails
#9952	FTS: submit client should ask for MyProxy password twice
#9959	FTS: scripts to check SRM timings, etc
#9967	FTS: drop Schema for old versions missing in rpm
#9969	Java and Perl FTS APIs not installed in WN

Bugs fixed in this or previous releases but not yet tested:

Bug Number	Description
#6931	FTS: Should use user proxy for transfer
#6932	FTS: Should support Oracle for its back end database
#6933	FTS: Should support Channel management
#7113	FTS: glite-url-copy - if SRM.put fails, the code loops forever
#7352	FTS: forked glite-url-copy should chdir somewhere it can write
#7354	FTS: glite_config_service waits for all child transfer processes to finish before stopping
#7358	FTS: channel allocation fails if the SURL doesn't contain a port number
#7361	FTS: transfer agent should detect a bad Oracle connection, and try to reconnect
#7506	FTS: requirement 1.2 should be met: proper retry
#7508	FTS: now needs to implement the protocol option passing as defined in interface
#7517	FTS: rename FilePlacement ot FileTransfer, fps -> fts
#7568	FTS: url-copy sometimes core dumps if SURL format is bad
#7581	FTS: add support for error categories
#7594	FTS: should migrate all components to FileTransfer interface
#7595	FTS: Need to test installation of -fts module
#7596	FTS: Commandline tools should be tested fully against service
#8333	FTS: service is not able to parse correctly the "grid-mapfile" with the administrators of the service
#8336	FTS: testing methods do not work
#8373	FTS: Certification with dCache, DPM and new castor SRM
#8376	FTS: per transfer logs need better management
#8486	FTS: submit commandline doesn't accept myproxy password
#8503	FTS: detailed transfer-status commandline doesn't work correctly
#8504	FTS: failure message for channel-list could be better
#8538	FTS: channel-set -S Active does not work.
#8588	FTS: CLI transfer-submit option -g produces Out of memory.
#8589	FTS: transfer-submit -f FILE doesn't recognize newline as delimiter
#8592	FTS: command-line and java service mispelling of Canceling and Canceled
#8690	FTS: channel management setStatusForHeldJobs has problems
#8701	FTS: server should accept equalsIgnoreCase("gridftp") for the transfer params
#8702	FTS: default portal logging should be DEBUG
#8711	FTS: CLI status cannot parse Waiting or Hold states
#8713	FTS: when FTS is misconfigured, the client gets obscure error messages
#8715	FTS: calls to the service should be logged uniformly
#8783	FTS: agents EnableHold property is not read correctly
#8787	FTS: FTS alone should not transition to FinishedDirty
#8795	FTS: any glite-transfer-channel-set command sets the channel Active
#8826	FTS: if setDone fails on the destination, glite-url-copy flags as successful
#8878	FTS: Occassional infinite loops in glite-url-copy process
#8884	FTS: if SRMs hang, we should timeout
#8911	FTS: often the number of bytes transferred is incorrect in the DB
#8913	FTS: channel-set CLI should shoud allow Halted, Drain and Stopped
#8916	FTS: getVersion gives strange error
#8966	FTS: srm failures are not always detected and a NULL TURL is returned
#8968	FTS: if SRM get or put retries reach the max timeout, it didn't report as failed
#8969	FTS: don't fail hard if SRM putdone fails on source
#9032	FTS: primary timeout doesn't work on FTS url-copy when SRM fails on putDone
#9036	FTS: the version reported by the CLI from the webservice is incorrect
#9166	FTS: No simple way to list all requestID's in a state
#9181	FTS: service restart does not always restart FTS agent
#9191	FTS: CLI tools need timeouts
#9200	FTS: glite-url-copy uses both jar and tar
#9201	FTS: glite-url-copy-cleanlog usage message issues
#9207	FTS: glite-url-copy arguments are order sensitive
#9223	FTS: glite-url-copy print out perf markers in blocking mode?
#9224	FTS: Can't copy large files to dCache
#9232	FTS: glite-url-copy doesn't timeout if file never transitions from pending
#9236	FTS: glite-url-copy segfaults on readdir if directory does not exist.
#9237	FTS: glite-transfer-list should have -a option
#9240	FTS: glite-url-copy does not distinguish in FINAL message on timeouts
#9241	FTS: Put request ID from SRM into log
#9309	FTS: dCache returns error on gridftp call, but glite-url-copy doesn't see it?
#9310	FTS: two FINAL messages if advisoryDelete fails against dCache
#9322	FTS: Formatting could be better on new log messages
#9398	FTS: mysterious failure on PUT when it apparently completed OK
#9399	FTS: blocking glite-url-copy segfaults or buffer overrruns sometimes on srm-putdone error
#9625	FTS: fails if double slash in filename

6. Bugs closed since last release

This release fixes the following bugs and issues. Bug numbers refer to the gLite Bug Tracking system database hosted on the CERN Savannah system at https://savannah.cern.ch/bugs/?group=jra1mdw

Bug number	Description
#8366	FTS: Transfer Agent for a channel is per-VO
#8386	FTS: No inter-VO scheduling
#9167	FTS: FTS forks off many many processes
#9259	FTS: allow channel level canceling
#9552	FTS: glite-url-copy dies when > 10 streams to dCache targets
#9718	FTS must support srmcopy for LCG experiments