Latest revision as of 12:05, 3 April 2013

RAL Tier1 Operations Report for 3rd April 2013

• On Thursday morning, 21st March around 08:40 to 09:00 there was a networking problem that caused some transitory problems for the Tier1. The effect was seen in a single SUM test failure for Atlas.
• Overnight Monday/Tuesday 25/26 March there was a failure of one of the three top BDII nodes. It was removed from the alias the following morning.
• On Tuesday 26th Mar high CPU usage on the site firewall caused intermittent network problems affecting the Tier1 from ~08:00 – ~09:45. Services again experienced transitory failures. There were FTS failures & SAM test failures.
• On Thursday afternoon, around 16:00, an operational error led to a networking break that caused some transitory problems for the Tier1.
• Services ran well over the Easter weekend. There were a couple of problems that did not affect front line services although one of the perfsonar network monitoring systems went down.

• GDSS446 (AtlasDataDisk D1T0) was taken out of service after reporting FSProbe errors yesterday evening (2nd April). A disk drive in the server has been replaced and it was returned to service around 12:25 today.

• The uplink to the UKLight Router is running on a single 10Gbit link, rather than a pair of such links.
• There have been intermittent problems over the past few weeks with the start rate for batch jobs. A script has been introduced to regularly check for this and take action to minimise its effects.
• The problem LHCb and Atlas jobs failing due to long job set-up times remains. A different version of CVMFS has been installed as a test and investigations continue.
• The testing of FTS3 is continuing. (This runs in parallel with our existing FTS2 service).
• We are participating in xrootd federated access tests for Atlas.
• Test batch queue with five SL6/EMI-2 worker nodes and own CE in place.
• There is an outstanding problem (and GGUS ticket) affecting the cerificate on the MyProxy server.

• None

• Updating disk controller firmware on Clustervision '11 batch of disk servers ongoing.
• Kernel/errata updates and removal of AFS software (as opposed to just disabling) being done across worker nodes.
• Removal of EMI-1 WMS systems (WMS01,02,03). (Disabled in GOC DB).

• This evening (Wed 3rd March 18:00 - 23:59 BST) Emergency maintenance in Geneva affecting both the main and backup links to CERN. No outage is expected during this maintenance. Services are considered at risk only.

• Next Tuesday (9th April) morning a networking update will cause a couple of short breaks in external connectivity as switches are rebooted.
• One of the disk arrays hosting the LFC/FTS/3D databases has given some errors and an intervention will be necessary requiring a stop of these services. We are checking how long this will take ahead of making an announcement.
• A program of updating the disk controller firmware in the 2011 Clustervision batch of disk servers is ongoing.

Listing by category:

• Databases:
  • Switch LFC/FTS/3D to new Database Infrastructure.
• Castor:
  • Upgrade to version 2.1.13
• Networking:
  • Single link to UKLight Router to be restored as paired (2*10Gbit) link.
  • Update core Tier1 network and change connection to site and OPN including:
    • Install new Routing layer for Tier1
    • Change the way the Tier1 connects to the RAL network.
    • These changes will lead to the removal of the UKLight Router.
  • Addition of caching DNSs into the Tier1 network.
• Grid Services
  • Upgrade of other EMI-1 components (APEL, UI) under investigation.
• Infrastructure:
  • Intervention required on the "Essential Power Board" & Remedial work on three (out of four) transformers.
  • Remedial work on the BMS (Building Management System) due to one its three modules being faulty.
  • Electrical safety check (will require some downtime).

There were no unscheduled entries in the GOC DB for the last fortnight.