Difference between revisions of "RAL Tier1 weekly operations castor 11/05/2015"

Latest revision as of 15:27, 11 May 2015

ATLAS draining complete
SL6 disk server config is production ready, but will not be deployed until DB 11.2.0.4 upgrade is done.
2 SL6 tape servers now running in production.
Testing CASTOR rebalancer on preproduction.

CMS CASTOR issues affecting batch farm efficiency - several areas of investigation / mitigation underway
Seeing a high volume of SRM DB file dup errors, Rob is currently investigating (trying to reproduce with FTS transfers)
Retrieval errors from facilities castor - a number of similar incidents seen in last few weeks
log rotate issue - transfer manager logs for rebalancing on CASTOR stagers
GDSS757 cmsDisk / 763 atlasStripInput - new motherboards, fabric acceptance test complete, ready for deployment.
Elastic tape data corruption - CASTOR team assisting with extraction of data for review
A higher number of checksum errors (Alice) investigation ongoing

OLDER>>>>
Deadlocks on atlas stager DB (post 2.1.14-15) - not serious but investigation ongoing
Issue with missing files in LHCb – race condition reported to CERN
Atlas are putting files (sonar.test files) into un-routable paths - this looks like an issue with the space token used. Brian / Rob actions below.
There is a problem with latest gfal libraries - not a new problem but Atlas are starting to exercise functionality and identifying these issues
storageD retrieval from castor problems - investigation ongoing
castor functional test on lcgccvm02 causing problems - Gareth reviewing.

grid ftp bug in SL6 - stops any globus copy if a client is using a particular library. This is a show stopper for SL6 on disk server.

Tasks

Create new small VOs for DiRAC (and LIGO?)
Bruno working on SL6 disk servers
Provide new VM? to provide castor client functionality to query the backup DBs
Plan to ensure PreProd represents production in terms of hardware generation are underway
Switch from admin machines: lcgccvm02 to lcgcadm05
Correct partitioning alignment issue (3rd CASTOR partition) on new castor disk servers
Intervention to upgrade CASTOR DBs to Oracle 11.2.0.4

Interventions

Staff absence/out of the office:
- Juan out from Tues
- Jens out Monday/Tues
- Shaun not available on Wed

Rob and Matt to escalate CASTOR slowness to Massimo at CERN
Shaun to send plots to Matt to support above action
Rob to book meeting to discuss possible workarounds/investigations for CMS issue - xroot timeout / server(read-ahead) or Network tuning / CASTOR bug / deploying more disk servers
Rob to try and reproduce SRM DB Dups issue with FTS transfers
Rob - change checksum validator so it maintains case when displaying
Bruno to document processes to control services previously controlled by puppet
Gareth to arrange meeting castor/fab/production to discuss the decommissioning procedures
Gareth to investigate providing checks for /etc/noquatto on production nodes & checks for fetch-crl

Brian - On Wed, tell Tim if he can start repacking mctape (Done)
Rob - talk with Shaun to see if its possible to reject anything that does not have a spacetoken (answer: this is not a good idea)
Brian - to discuss unroutable files / spacetokens at at DDM meeting on Tuesday (Done)
Rob to pick up DB cleanup change control (Done)
Chris/Rob to arrange a meeting to discuss CMS performance/xroot issues (is performance appropriate, if not plan to resolve) - inc. Shaun, Rob, Brian, Gareth (Potential fix implemented)

@@ Line 60: / Line 60: @@
 * Rob to book meeting to discuss possible workarounds/investigations for CMS issue - xroot timeout / server(read-ahead) or Network tuning / CASTOR bug / deploying more disk servers
 * Rob to try and reproduce SRM DB Dups issue with FTS transfers
-* Rob - revisit deployment plans (esp people in office towards the end of March)
 * Rob - change checksum validator so it maintains case when displaying
 * Bruno to document processes to control services previously controlled by puppet