Difference between revisions of "Tier1 Operations Report 2018-08-20"

From GridPP Wiki
Jump to: navigation, search
(→‎RAL Tier1 Operations Report for 20th August 2018)
(→‎)
 
(17 intermediate revisions by one user not shown)
Line 22: Line 22:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Current operational status and issues
 
|}
 
|}
−
* Requirement to purchase new SIM (an associated contract), to replace the failed EE SIM in the SMS system.
+
* The new siny O2 SIM for the SMS service has been delivered and installed
 
<!-- ***********End Current operational status and issues*********** ----->
 
<!-- ***********End Current operational status and issues*********** ----->
 
<!-- *************************************************************** ----->
 
<!-- *************************************************************** ----->
Line 41: Line 41:
 
! Comments
 
! Comments
 
|-
 
|-
−
| gdss690
+
| -
−
| LHCb
+
| -
−
| lhcbDst
+
| -
−
| d1t0
+
| -
−
| Back in production RO..
+
| -
 
|}
 
|}
 
<!-- ***************************************************** ----->
 
<!-- ***************************************************** ----->
Line 81: Line 81:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Limits on concurrent batch system jobs.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Limits on concurrent batch system jobs.
 
|}
 
|}
−
* CMS Multicore 550
+
* GROUP_CMS_LIMIT = 4000
 +
* GROUP_ATLAS_LIMIT = 8000
 
<!-- ******************End Limits On Batch System Jobs***************** ----->
 
<!-- ******************End Limits On Batch System Jobs***************** ----->
 
<!-- ****************************************************************** ----->
 
<!-- ****************************************************************** ----->
Line 115: Line 116:
 
! Reason
 
! Reason
 
|-
 
|-
−
| ECHO
+
| -
−
| 25871
+
| -
−
| UNSCHEDULED OUTAGE
+
| -
−
| OUTAGE
+
| -
−
| 10/08/2018 20:00
+
| -
−
| 16/08/2018 12:00
+
| -
−
| 7 days
+
| -
−
| Major problems with the ECHO instance at RAL
+
| -
 
|}
 
|}
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- **********************End GOC DB Entries************************** ----->
Line 147: Line 148:
 
! Reason
 
! Reason
 
|-
 
|-
−
| LFC
+
| -
−
| 25853
+
| -
−
| SCHEDULED OUTAGE
+
| -
−
| OUTAGE
+
| -
−
| 15/08/2018 08:00
+
| -
−
| 15/08/2018 17:00
+
| -
−
| 1 days
+
| -
−
| Major upgrade of systems
+
| -
−
|-
+
−
| ECHO
+
−
| 25871
+
−
| UNSCHEDULED OUTAGE
+
−
| OUTAGE
+
−
| 10/08/2018 20:00
+
−
| 16/08/2018 12:00
+
−
| 7 days
+
−
| Major problems with the ECHO instance at RAL
+
 
|}
 
|}
 +
* No ongoing downtime
 +
* No downtime scheduled in the GOCDB for next 2 weeks
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- **********************End GOC DB Entries************************** ----->
 
<!-- ****************************************************************** ----->
 
<!-- ****************************************************************** ----->
Line 207: Line 201:
 
! Subject
 
! Subject
 
! Scope
 
! Scope
 +
|-
 +
| style="background-color: green;" | 136757
 +
| mice
 +
| in progress
 +
| less urgent
 +
| 17/08/2018
 +
| 21/08/2018
 +
| Other
 +
| Missing lsc files for mice VO on lfc.gridpp.rl.ac.uk ?
 +
| EGI
 
|-
 
|-
 
| style="background-color: green;" | 136701
 
| style="background-color: green;" | 136701
Line 213: Line 217:
 
| very urgent
 
| very urgent
 
| 14/08/2018
 
| 14/08/2018
−
| 14/08/2018
+
| 21/08/2018
 
| File Transfer
 
| File Transfer
 
| background of transfer errors
 
| background of transfer errors
−
| WLCG
 
−
|-
 
−
| style="background-color: green;" | 136665
 
−
| cms
 
−
| in progress
 
−
| urgent
 
−
| 11/08/2018
 
−
| 14/08/2018
 
−
| CMS_SAM tests
 
−
| T1_UK_RAL is down > 12h
 
−
| WLCG
 
−
|-
 
−
| style="background-color: green;" | 136563
 
−
| cms
 
−
| in progress
 
−
| urgent
 
−
| 06/08/2018
 
−
| 06/08/2018
 
−
| CMS_Data Transfers
 
−
| Possibly corrupted files at RAL
 
 
| WLCG
 
| WLCG
 
|-
 
|-
 
| style="background-color: green;" | 136366
 
| style="background-color: green;" | 136366
 
| mice
 
| mice
−
| waiting for reply
+
| in progress
 
| less urgent
 
| less urgent
 
| 25/07/2018
 
| 25/07/2018
−
| 08/08/2018
+
| 20/08/2018
 
| Local Batch System
 
| Local Batch System
 
| Remove MICE Queue from RAL T1 Batch
 
| Remove MICE Queue from RAL T1 Batch
 
| EGI
 
| EGI
 
|-
 
|-
−
| style="background-color: green;" | 136358
+
| style="background-color: orange;" | 136199
−
| cms
+
−
| on hold
+
−
| urgent
+
−
| 25/07/2018
+
−
| 13/08/2018
+
−
| CMS_Facilities
+
−
| T1_UK_RAL WN-xrootd-access failure
+
−
| WLCG
+
−
|-
+
−
| style="background-color: yellow;" | 136199
+
 
| lhcb
 
| lhcb
 
| in progress
 
| in progress
Line 273: Line 247:
 
| top priority
 
| top priority
 
| 10/07/2018
 
| 10/07/2018
−
| 13/08/2018
+
| 21/08/2018
 
| CMS_AAA WAN Access
 
| CMS_AAA WAN Access
 
| Issues reading files at T1_UK_RAL_Disk
 
| Issues reading files at T1_UK_RAL_Disk
Line 312: Line 286:
 
! Scope
 
! Scope
 
|-
 
|-
−
| style="background-color: green;" | 136537
+
| 136655
−
| ops
+
| lhcb
 
| verified
 
| verified
 
| less urgent
 
| less urgent
−
| 03/08/2018
+
| 10/08/2018
−
| 06/08/2018
+
| 15/08/2018
−
| Operations
+
| File Access
−
| [Rod Dashboard] Issue detected : org.bdii.GLUE2-Validate@site-bdii.gridpp.rl.ac.uk
+
| Missing File At RAL
−
| EGI
+
| WLCG
 
|-
 
|-
−
| 136359
+
| 136460
−
| none
+
| cms
 
| closed
 
| closed
−
| top priority
+
| urgent
−
| 25/07/2018
+
| 30/07/2018
−
| 08/08/2018
+
| 15/08/2018
−
| Other
+
| CMS_Data Transfers
−
| This TEST ALARM has been raised for testing GGUS alarm work flow after a new GGUS release.
+
| Transfers failing to RAL_Buffer
 
| WLCG
 
| WLCG
 
|-
 
|-
−
| 136270
+
| 136427
−
| cms
+
| atlas
 
| closed
 
| closed
 
| urgent
 
| urgent
−
| 22/07/2018
+
| 28/07/2018
−
| 07/08/2018
+
| 13/08/2018
−
| CMS_Facilities
+
| File Transfer
−
| T1_UK_RAL xrootd access failures
+
| UK RAL-LCG2: Transfer errors as destination
 
| WLCG
 
| WLCG
 
|-
 
|-
−
| 134685
+
| 136408
−
| dteam
+
| cms
−
| solved
+
| closed
−
| less urgent
+
| urgent
−
| 23/04/2018
+
| 27/07/2018
−
| 08/08/2018
+
| 15/08/2018
−
| Middleware
+
| CMS_Data Transfers
−
| please upgrade perfsonar host(s) at RAL-LCG2 to CentOS7
+
| missing files at RAL
−
| EGI
+
| WLCG
 
|}
 
|}
 
<!-- **********************End Availability Report************************** ----->
 
<!-- **********************End Availability Report************************** ----->
Line 381: Line 355:
 
! Comments
 
! Comments
 
|-
 
|-
−
| 2018-08-06
+
| 2018-08-13
 
| 100
 
| 100
 
| 100
 
| 100
−
| 99
+
| style="background-color: red;" | 0
 
| 100
 
| 100
 
| 100
 
| 100
Line 390: Line 364:
 
|  
 
|  
 
|-
 
|-
−
| 2018-08-07
+
| 2018-08-14
 
| 100
 
| 100
 
| 100
 
| 100
−
| style="background-color: orange;" | 94
+
| style="background-color: red;" | 0
 
| 100
 
| 100
 
| 100
 
| 100
Line 399: Line 373:
 
|  
 
|  
 
|-
 
|-
−
| 2018-08-08
+
| 2018-08-15
 
| 100
 
| 100
 
| 100
 
| 100
−
| style="background-color: red;" | 89
+
| style="background-color: red;" | 0
 
| 100
 
| 100
 
| 100
 
| 100
Line 408: Line 382:
 
|  
 
|  
 
|-
 
|-
−
| 2018-08-09
+
| 2018-08-16
 
| 100
 
| 100
 
| 100
 
| 100
−
| style="background-color: orange;" | 93
+
| style="background-color: red;" | 0
 
| 100
 
| 100
 
| 100
 
| 100
Line 417: Line 391:
 
|  
 
|  
 
|-
 
|-
−
| 2018-08-10
+
| 2018-08-17
 
| 100
 
| 100
 
| 100
 
| 100
−
| style="background-color: orange;" | 95
+
| style="background-color: red;" | 60
 
| 100
 
| 100
 
| 100
 
| 100
Line 426: Line 400:
 
|  
 
|  
 
|-
 
|-
−
| 2018-08-11
+
| 2018-08-18
 
| 100
 
| 100
 
| 100
 
| 100
−
| style="background-color: red;" | 31
 
 
| 100
 
| 100
 
| 100
 
| 100
 
| 100
 
| 100
−
| Caused by Echo outage
+
| 100
−
+
|
 
|-
 
|-
−
| 2018-08-12
+
| 2018-08-19
 
| 100
 
| 100
 
| 100
 
| 100
−
| style="background-color: red;" | 7
 
 
| 100
 
| 100
 
| 100
 
| 100
 
| 100
 
| 100
−
| Caused by Echo outage
+
| 100
 +
|
 
|-
 
|-
−
| 2018-08-13
+
| 2018-08-20
 
| 100
 
| 100
 
| 100
 
| 100
−
| style="background-color: red;" | 0
 
 
| 100
 
| 100
 
| 100
 
| 100
 
| 100
 
| 100
−
| Caused by Echo outage
+
| 100
 +
|
 
|}
 
|}
  
Line 471: Line 444:
 
! Day !! Atlas HC !! CMS HC !! Comment
 
! Day !! Atlas HC !! CMS HC !! Comment
 
|-
 
|-
−
| 2018-08-06 || 100 ||  100 ||   
+
| 2018-08-13 || 0 ||  0 ||   
 
|-
 
|-
−
| 2018-08-07 || 100 || 99 ||  
+
| 2018-08-14 || 0 || 0 ||  
 
|-
 
|-
−
| 2018-08-08 || style="background-color: orange;" | 91 || 98 ||  
+
| 2018-08-15 || 0 || 0 ||  
 
|-
 
|-
−
| 2018-08-09 || 100 || style="background-color: orange;" | 93 ||  
+
| 2018-08-16 || 0 || 0 ||  
 
|-
 
|-
−
| 2018-08-10 || 98 || style="background-color: red;" | 87||  
+
| 2018-08-17 || 76 || 60||  
 
|-
 
|-
−
| 2018-08-11 || style="background-color: red;" | 67 || style="background-color: orange;" | 91 ||  
+
| 2018-08-18 || 100 || 100||  
 
|-
 
|-
−
| 2018-08-12 || 100 || 97 ||  
+
| 2018-08-19 || 100 || 100 ||  
 
|-
 
|-
−
| 2018-08-13 || style="background-color: red;" | 0 || 100 ||  
+
| 2018-08-20 || 100 || 100 ||  
 
|-
 
|-
 
|}  
 
|}  
Line 499: Line 472:
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notes from Meeting.
 
| style="background-color: #b7f1ce; border-bottom: 1px solid silver; text-align: center; font-size: 1em; font-weight: bold; margin-top: 0; margin-bottom: 0; padding-top: 0.1em; padding-bottom: 0.1em;" | Notes from Meeting.
 
|}
 
|}
−
*
+
* The recent problems with Echo were discussed. A summary is also in the Operations report. Several points:
 +
* Current situation is that we have “production” access running
 +
* Atlas and CMS have upper limits on batch jobs about equal to pledge.
 +
* The additional memory for the Dell storage nodes has arrived. This will be added – at least initially in a rolling upgrade. However, it is expected this may take up to something like 2 weeks. (At which point we expect to be able to give full access to Echo).  Discussions on the best way to do the memory upgrades is ongoing.
 +
* Before the problem we had noted that the LHCb files came from Castor in a “bursty” way. It was suggested last week that we limit the FTS to smooth out this burstiness.
 +
* When discussing GGUS tickets: LHCb are seeing some file transfer failures between worker nodes and Castor. This to be escalated.
 +
* The requirement for us to be able to easily contact all users of Echo in the event of a problem was noted.

Latest revision as of 13:24, 22 August 2018

RAL Tier1 Operations Report for 20th August 2018

Review of Issues during the week 13th August to the 20th August 2018.
  • The upgrade of Echo was completed successfully on Thursday (16/8/18), with a greatly reduced memory usage. The cluster was allowed to recover overnight. Everything appeared to be working well on the Friday (17/8/18), and there is currently no evidence of data loss. We therefore ended the downtime at Friday 12:00 UTC (17/8/18). As a precaution for the weekend, we limited the ATLAS (and CMS), quota on our batch farm to 50% of its nominal amount. Assuming we encounter no problems we intend to lift this on Monday (20/8/18).
Current operational status and issues
  • The new siny O2 SIM for the SMS service has been delivered and installed
Resolved Castor Disk Server Issues
Machine VO DiskPool dxtx Comments
- - - - -
Ongoing Castor Disk Server Issues
Machine VO DiskPool dxtx Comments
gdss747 Atlas atlasStripInput d1t0 Currently in intervention.
Limits on concurrent batch system jobs.
  • GROUP_CMS_LIMIT = 4000
  • GROUP_ATLAS_LIMIT = 8000
Notable Changes made since the last meeting.
  • None.
Entries in GOC DB starting since the last report.
Service ID Scheduled? Outage/At Risk Start End Duration Reason
- - - - - - - -
Declared in the GOC DB
Service ID Scheduled? Outage/At Risk Start End Duration Reason
- - - - - - - -
  • No ongoing downtime
  • No downtime scheduled in the GOCDB for next 2 weeks
Advanced warning for other interventions
The following items are being discussed and are still to be formally scheduled and announced.

Listing by category:

  • Castor:
    • Update systems to use SL7 and configured by Quattor/Aquilon. (Tape servers done)
    • Move to generic Castor headnodes.
  • Internal
    • DNS servers will be rolled out within the Tier1 network.
Open

GGUS Tickets (Snapshot taken during morning of the meeting).

Request id Affected vo Status Priority Date of creation Last update Type of problem Subject Scope
136757 mice in progress less urgent 17/08/2018 21/08/2018 Other Missing lsc files for mice VO on lfc.gridpp.rl.ac.uk ? EGI
136701 lhcb in progress very urgent 14/08/2018 21/08/2018 File Transfer background of transfer errors WLCG
136366 mice in progress less urgent 25/07/2018 20/08/2018 Local Batch System Remove MICE Queue from RAL T1 Batch EGI
136199 lhcb in progress very urgent 18/07/2018 07/08/2018 File Transfer Lots of submitted transfers on RAL FTS WLCG
136028 cms in progress top priority 10/07/2018 21/08/2018 CMS_AAA WAN Access Issues reading files at T1_UK_RAL_Disk WLCG
124876 ops in progress less urgent 07/11/2016 23/07/2018 Operations [Rod Dashboard] Issue detected : hr.srce.GridFTP-Transfer-ops@gridftp.echo.stfc.ac.uk EGI
GGUS Tickets Closed Last week
Request id Affected vo Status Priority Date of creation Last update Type of problem Subject Scope
136655 lhcb verified less urgent 10/08/2018 15/08/2018 File Access Missing File At RAL WLCG
136460 cms closed urgent 30/07/2018 15/08/2018 CMS_Data Transfers Transfers failing to RAL_Buffer WLCG
136427 atlas closed urgent 28/07/2018 13/08/2018 File Transfer UK RAL-LCG2: Transfer errors as destination WLCG
136408 cms closed urgent 27/07/2018 15/08/2018 CMS_Data Transfers missing files at RAL WLCG

Availability Report

Target Availability for each site is 97.0% Red <90% Orange <97%
Day Atlas Atlas-Echo CMS LHCB Alice OPS Comments
2018-08-13 100 100 0 100 100 100
2018-08-14 100 100 0 100 100 100
2018-08-15 100 100 0 100 100 100
2018-08-16 100 100 0 100 100 100
2018-08-17 100 100 60 100 100 100
2018-08-18 100 100 100 100 100 100
2018-08-19 100 100 100 100 100 100
2018-08-20 100 100 100 100 100 100
Hammercloud Test Report
Target Availability for each site is 97.0% Red <90% Orange <97%
Day Atlas HC CMS HC Comment
2018-08-13 0 0
2018-08-14 0 0
2018-08-15 0 0
2018-08-16 0 0
2018-08-17 76 60
2018-08-18 100 100
2018-08-19 100 100
2018-08-20 100 100

Key: Atlas HC = Atlas HammerCloud (Queue RAL-LCG2_UCORE, Template 841); CMS HC = CMS HammerCloud

Notes from Meeting.
  • The recent problems with Echo were discussed. A summary is also in the Operations report. Several points:
  • Current situation is that we have “production” access running
  • Atlas and CMS have upper limits on batch jobs about equal to pledge.
  • The additional memory for the Dell storage nodes has arrived. This will be added – at least initially in a rolling upgrade. However, it is expected this may take up to something like 2 weeks. (At which point we expect to be able to give full access to Echo). Discussions on the best way to do the memory upgrades is ongoing.
  • Before the problem we had noted that the LHCb files came from Castor in a “bursty” way. It was suggested last week that we limit the FTS to smooth out this burstiness.
  • When discussing GGUS tickets: LHCb are seeing some file transfer failures between worker nodes and Castor. This to be escalated.
  • The requirement for us to be able to easily contact all users of Echo in the event of a problem was noted.
Retrieved from "https://www.gridpp.ac.uk/w/index.php?title=Tier1_Operations_Report_2018-08-20&oldid=18377"