Tier1 Operations Report 2018-10-15

From GridPP Wiki
Jump to: navigation, search

RAL Tier1 Operations Report for 15th October 2018

Review of Issues during the week 8th October 2018 to the 15th October 2018.
  • No major issues of note to report for this week.
  • Occasional repeat of a CMS error – “500-globus_xio: System error in bind: Address already in use.” - This is not a major issue but does impact the SAM statistics.
Current operational status and issues
  • As an advisory Tier-1 is continuing with the current round of security patching (CVE-2018-14634, CVE-2018-8897 – SL7 and SL6 respectively). This involves all Tier-1 services but the risk of impact on availability is considered to be minimal and as such no declaration of service “At Risk” or “Outage” will be called.
Resolved Castor Disk Server Issues
Machine VO DiskPool dxtx Comments
gdss753 Atlas atlasStripInput d1t0 Decommissioned.
Ongoing Castor Disk Server Issues
Machine VO DiskPool dxtx Comments
gdss747 Atlas atlasStripInput d1t0 Currently in intervention.
Limits on concurrent batch system jobs.
  • None currently enforced.
Notable Changes made since the last meeting.
  • None.
Entries in GOC DB starting since the last report.
Service ID Scheduled? Outage/At Risk Start End Duration Reason
- - - - - - - -
Declared in the GOC DB
Service ID Scheduled? Outage/At Risk Start End Duration Reason
- - - - - - - -
  • No ongoing downtime
  • No downtime scheduled in the GOCDB for next 2 weeks
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.

Listing by category:

  • Castor:
    • Disk only storage will end in Castor. A new single tape-only Castor instance (called WLCGTape) is being tested. This is using generic Catsor headnodes on SL7 configured by Quattor/Aquilon with a slightly newer Castor version.
    • Update systems to use SL7 and configured by Quattor/Aquilon. (Tape servers done)
  • Internal
    • DNS servers will be rolled out within the Tier1 network.
Open

GGUS Tickets (Snapshot taken during morning of the meeting).

Request id Affected vo Status Priority Date of creation Last update Type of problem Subject Scope
137650 cms waiting for reply urgent 09/10/2018 15/10/2018 CMS_AAA WAN Access Low HC xrootd success rates at T1_UK_RAL WLCG
137195 ops in progress less urgent 14/09/2018 15/10/2018 Operations [Rod Dashboard] Issues detected at RAL-LCG2 EGI
137153 t2k.org in progress urgent 12/09/2018 10/10/2018 Data Management - generic LFC entry has file size 0, preventsw registering of additional replicas EGI
136701 lhcb waiting for reply very urgent 14/08/2018 10/10/2018 File Transfer background of transfer errors WLCG
136199 lhcb waiting for reply very urgent 18/07/2018 11/10/2018 File Transfer Lots of submitted transfers on RAL FTS WLCG
124876 ops in progress less urgent 07/11/2016 08/10/2018 Operations [Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk EGI
GGUS Tickets Closed Last week
Request id Affected vo Status Priority Date of creation Last update Type of problem Subject Scope
137723 cms solved urgent 14/10/2018 15/10/2018 CMS_Data Transfers PhEDEx component Agent Watchdog donw at T1_UK-RAL WLCG
137699 dteam verified top priority 11/10/2018 15/10/2018 Monitoring Test of RAL-LCG2 Alarm Ticket Handling WLCG
137675 ops verified less urgent 10/10/2018 11/10/2018 Operations [Rod Dashboard] Issue detected : org.nordugrid.ARC-CE-ARIS@arc-ce04.gridpp.rl.ac.uk EGI
137634 cms solved urgent 08/10/2018 10/10/2018 CMS_Data Transfers Transfers failing from T1_UK_RAL_Disk to METU WLCG
137619 cms solved urgent 07/10/2018 08/10/2018 CMS_AAA WAN Access T1_UK_RAL xrootd read failures WLCG
137434 cms closed urgent 26/09/2018 11/10/2018 CMS_AAA WAN Access xrootd host not reachable at T1_UK_RAL WLCG
137391 atlas solved urgent 25/09/2018 10/10/2018 Network problem UK RAL-LCG2 transfer errors with Communication error on send WLCG
137151 t2k.org closed urgent 12/09/2018 08/10/2018 File Transfer Cannot upload files to grid an register replica EGI
136757 mice closed less urgent 17/08/2018 09/10/2018 Other Missing lsc files for mice VO on lfc.gridpp.rl.ac.uk ? EGI
136028 cms closed top priority 10/07/2018 12/10/2018 CMS_AAA WAN Access Issues reading files at T1_UK_RAL_Disk WLCG

Availability Report

Day Atlas Atlas-Echo CMS LHCB Alice
2018-10-08 100 100 100 100 100
2018-10-09 100 100 99 100 100
2018-10-10 100 100 98 100 100
2018-10-11 100 100 98 100 100
2018-10-12 100 100 100 100 100
2018-10-13 100 100 100 100 100
2018-10-14 100 100 100 100 100
2018-10-15 100 100 100 100 100
Hammercloud Test Report
Target Availability for each site is 97.0% Red <90% Orange <97%
Day Atlas HC CMS HC Comment
2018-08-01 100 98
2018-08-02 100 99
2018-08-03 100 99
2018-08-04 100 99
2018-08-05 100 98
2018-08-06 100 98
2018-08-07 100 96
2018-08-08 100 96

Key: Atlas HC = Atlas HammerCloud (Queue RAL-LCG2_UCORE, Template 841); CMS HC = CMS HammerCloud

Notes from Meeting.
Retrieved from "https://archive.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2018-10-15&oldid=18727"