Difference between revisions of "Operations Bulletin 040515"

Tuesday 5th May

• It is a CMS week this week.
• A [ https://indico.cern.ch/event/319821/ pre-GDB on batch systems is taking place next Tuesday 12th May]. More T2 participation is sought. Still need to define T2 GDB rep.
• CHEP'15 proceedings submissions due byMay 17th.

Tuesday 28th April

• May pre-GDB will now be half a day (various contributors away). Agenda soon.
• Raul reports major problem upgrading Torque from 2.xx to 4.2 and warn others (see ticket).
• Please keep the batch systems table updated with any changes at your site.
• What is the status of the WN tarball?
• Alexander Dibbo joins the T1 team to work in the Grid Services team (eg. (hypervisors, updating WN Grid software….).
• EGI is advertising FitSM Foundation training in IT Service Management, 21-22 May – Lisbon.
• Several sites noted as having APEL sync problems in March: UCL, RHUL, and ECDF. The sites were asked to trial new documentation on republishing. Any feedback?
• The final T2 A&R reports for March have been produced.
• Tom is collecting information on GridPP talks and papers. Please use these forms to alert him: Publications. [ https://docs.google.com/forms/d/1_izxIHlgbKFePlH6AcSvO5rbM9ypZ9LqU5c02Ri33-I/viewform Talks].
• EGI RC OLA and RP OLA Reports for March 2015 are now available.

Tuesday 28th April

• A WLCG ops coordination meeting took place on 23rd April. Minutes.
• News: No special issues.
• Workshop: Feedback being compiled.
• Middleware news: NTR.
• Baselines: FTS3 now at v3.2.33. Improves REST API performance.
• T0 & T1 services: Minor dCache upgrades.
• T0 news: NTR
• T1 feedback: NTR
• T2 feedback: NTR
• ALICE: New job record of 80k+. Taking advantage of resource availability. Some CASTOR file access instabilities for re-reco jobs.
• ATLAS: NTR
• CMS: DIGI-RECO of Upgrade MC at T1s. At EOS unmerged area quota limit. Possible CASTOR connections saturation issue.
• LHCb: MC production and user analysis progressing. RAL network problems 8/9th April.
• gLExec: ATLAS campaign at 61 of 94 sites. RAL/RALPP issues thought to be ARC CE/Condor related (TW-FTT switched to this setup and problems appeared there too).
• SHA-2: Old VOMS server aliases to be removed Tuesday 28th April.
• Machine/Job features: More site contacts for testing requested.
• MW readiness: Review actions from 18th March meeting. Work starting on visualization related to Package Reporter output. Next meeting 6th May 15:00 BST.
• MC: NTR
• IPv6: NTR
• Squid mon/HTTP proxy discovery: NTR.
• Network transfers: NTR
• HTTP deployment: NTR
• Next meeting 7th May.

Tuesday 21st April

• The next WLCG ops coordination meeting will be on Thursday 23rd. Jeremy will try to attend, but please could one or two others from core ops provide backup representation.

Tuesday 14th April

• No updates this week.

Tuesday 28th April

• Update to the Oracle database behind the LFC during today (Tuesday 28th April).
• We will turn off our remainining CREAM CEs on 5th May.
• The problems with our primary network router are still being followed up.

Tuesday 21st April

• Has there been any Tier-1 contact with DiRAC?
• Proposal to setup an 'other VOs' users list. GridPP-Users is too tied with WLCG projects.

Wednesday 15 April

• Backing up data from DiRAC to GridPP (tape)
• More case studies on supporting non-LHC VOs on GridPP: we have a lot of great stuff that can do great stuff - non-LHC VOs tend to have less regimented data models so maybe we need more case studies.

Wednesday 8 April

• Reports from HEPiX and Networkshop 43

Tuesday 7th April

• There was a DPM collaboration meeting last week. (Blog update)

Tuesday 31st March

• An [ https://indico.cern.ch/event/382643/ enlarged DPM development meeting] took place on Thursday 27th March.
• Any feedback from those who attended?

Tuesday 17th March

• An annual DPM collaboration board will take place in coming weeks. Are there issues that sites want raised (in relation to things like roadmap, concerns about DPM, requests etc.)?

Tuesday 21st April

• (Slight) Accounting delays seen for: UCL; Sheffield; QMUL & RALPP.

Tuesday 14th April

• APEL delays for UCL; Sheffield; RALPP and Bristol

• A reminder to keep updating the HEPSPEC06 tables.

• APEL status: An issue at Sheffield?

See the worst KeyDocs list for documents needing review now and the names of the responsible people.

Tuesday 21st April

• The Approved VOs document has been updated to take account of changes to the Ops Portal VOID cards.For SNOPLUS.SNOLAB.CA, the port numbers for voms02.gridpp.ac.uk and voms03.gridpp.ac.uk have both been updated from 15003 to 15503. Sites that support SNOPLUS.SNOLAB.CA should ensure that their configuration conforms to these settings: Approved VOs

• KeyDocs still need updating since agreements reached at last core ops meeting.

• Question of where to direct new VOs. EGI welcome page?

• New section in Wiki called "Project Management Pages".

The idea is to cluster all Self-Edited Site Tracking Tables
in here. Sites should keep entries in Current Activities
up to date. Once a Self-Edited Site Tracking Tables has
served its purpose, PM to move it to  Historical Archive 
or otherwise dispose of the table.

Tuesday 21st April

• There was an EGI ops meeting on Monday 20th.
• David updated the UK SL5 response.
• Please review the agenda/minutes.

Monday 9th March

• The agenda for February's EGI ops meeting is here. Minutes are here

• • APEL 1.4.0
    • Added Month and Year columns to primary key of CloudSummaries table in cloud schema.
  • DPM-Xrootd 3.5.2 is in EPEL stable - this is the first version of the component compatible with xrootd4
  • gLExec-wn - v. 1.2.3: lcmaps-plugins-c-pep 1.3.0-1 & mkgltempdir 0.0.5-1
    • "The lcmaps-plugins-c-pep-1.3.0-1 preferably needs the argus-pep-api-c-2.3.0. This version will be released into EMI & UMD repositories in a near future."
  • UMD 3.11.0 released on 16.02.2014, UMD 3.11.1 released on 4.03.2014
  • lcg-CA 1.62 noted with an intention to broadcast these as they occur as opposed to monthly.
  • EGI looking at the decommissioning of SL5, possibly by end of 2015, as a byproduct of adding CentOS 7 to UMD. NGIs to make a note if extended SL5 support is required.
  • Vincenzo Spinoso has joined EGI Ops team from NGI_IT. Vincenzo will chair EGI Ops.
  • Next meeting is April 20th.

Tuesday 31st March

• Glasgow Graphite/Grafana documentation: http://www.scotgrid.ac.uk/graphite/

Monday 7th December

• Meeting last Friday - agenda: https://indico.cern.ch/event/356853/ minutes: https://indico.cern.ch/event/356853/material/minutes/1.pdf
• This was the wrap-up meeting of the consolidation TF; the mailing list will remain extant for a while yet.

Tuesday 28th April

• Glasgow: A GLUE2 problem is transient and doesn't have a short-term solution (if the service status was checked a little more frequently it would help). Currently on hold. IC sometimes see this too.

• UCL: No change to the on-going situation. UCL has hopped from one downtime to another this week. Note – AM visiting UCL this week to setup VAC. Services will be decommissioned after this step.

Tuesday 21st April

• UCL have put themselves into a downtime until the 21st April. (Start of next week). Noted this in their outstanding tickets.
• Birmingham's availability has steadily recovered over the week - and the low availability ticket against them should be closable next week.

Tuesday 17th March

• Daniela has updated the [ https://www.gridpp.ac.uk/wiki/Staged_rollout_emi3 EMI-3 testing table]. Please check it is correct for your site. We want a clear view of where we are contributing.
• There is a middleware readiness meeting this Wednesday. Would be good if a few site representatives joined.
• Machine job features solution testing. Fed back that we will only commence tests if more documentation made available. This stops the HTC solution until after CHEP. Is there interest in testing other batch systems? Raul mentioned SLURM. There is also SGE and Torque.

Tuesday 17th February

• A message from Maria Dimou: A request to you and the sites you collaborate with, to please install the MW Package Reporter.

The current version, documented in the wiki , it is very easy to install and addresses all site and security requirements.

• Quick check on: 1. ARC CE readiness testing. 2. Machine job features testing.

References

• Staged Rollout pages (now separated into EMI1 & 2), and the page listing the deployed versions is extractable from the bdii, so they should all be reasonably up-to-date:
• http://www.hep.ph.ic.ac.uk/~dbauer/grid/staged_rollout.html
• http://www.hep.ph.ic.ac.uk/~dbauer/grid/staged_rollout_emi2.html
• http://www.hep.ph.ic.ac.uk/~dbauer/grid/state_of_the_nation.html

Tuesday 28th April

• SSL version issues with GridFTP at Bristol? Turns out the server doesn't like proxies that have not been decorated correctly.

Tuesday 21st April

• EGI Alert 'High' risk - Xen Vulnerability Hypervisor memory corruption due to x86 emulator flaw CVE-2015-2151 [EGI-ADV-20150415]

Tuesday 7th April

• EGI Trust Anchor release 1.63-1 (see announcement 30th March).

Tuesday 31st March

• EGI Advisory 'CRITICAL' RISK CVE-2015-1815 RedHat setroubleshoot [EGI-ADV-20150330]

• The EGI security dashboard.

- This includes notifying of (inter)national services that will have an outage in the coming weeks or will be impacted by work elsewhere. (Cross-check the Tier-1 update).

Tuesday 31st March

• Duncan has been giving the IPv6 nodes a push! See Steve's IPv6 transfers page.

Tuesday 10th March

• From the recent WLCG meeting, two slides (1 & 2) give the direction of the network monitoring and metrics progress: integration of perfSONAR event types into experiment monitoring and an architecture for data to get from RSV probes to client. Components described on slide 3.
• The next LHCOPN and LHCONE joint meeting will take place on Monday 1st and Tuesday 2nd of June 2015 in Berkeley (US) (hosted by LBL and ESnet).

Friday 1st May
The Bank Holiday weekend might muck up plans for a Ticket review this week. Just in case, some links!

Other VO Nagios page.

UK GGUS Tickets

Hope you all have a nice weekend!

Tuesday 17th February

• Another period where message brokers were temporarily unavailable seen yesterday. Any news on the last follow-up?

Tuesday 27th January

• Unscheduled outage of the EGI message broker (GRNET) caused a short-lived disruption to GridPP site monitoring (jobs failed) last Thursday 22nd January. Suspect BDII caching meant no immediate failover to stomp://mq.cro-ngi.hr:6163/ from stomp://mq.afroditi.hellasgrid.gr:6163/

• Blog about VO Nagios
• Oxford VO Nagios currently monitoring gridpp, pheno, t2k.org, snoplus.snolab.ca, vo.southgrid.ac.uk.

Tuesday 5th May

• We have a number of VOs to be removed. Dedicated follow-up meeting proposed.

Tuesday 28th April

• For SNOPLUS.SNOLAB.CA, the port numbers for voms02.gridpp.ac.uk and voms03.gridpp.ac.uk have both been updated from 15003 to 15503.

Tuesday 31st March

• LIGO are in need of additional support for debugging some tests.
• LSST now enabled on 3 sites. No 'own' CVMFS yet.

Monday 9th March

• SIXT VOMS parameters have changed.

Tuesday 17th February Two changes to approved VOs (https://www.gridpp.ac.uk/wiki/GridPP_approved_VOs)

• LSST uses port 15003 (had been 15002, clashing with dzero)
• t2k has included a note that it';s software is now distrinuted via CVMFS.

• Impact
  • Citation policy (https://www.gridpp.ac.uk/acknowledging.html)

Tuesday 24th February

• Next review of status today.

Tuesday 27th January

• Squids not in GOCDB for: UCL; ECDF; Birmingham; Durham; RHUL; IC; Sussex; Lancaster
• Squids in GOCDB for: EFDA-JET; Manchester; Liverpool; Cambridge; Sheffield; Bristol; Brunel; QMUL; T1; Oxford; Glasgow; RALPPD.

Tuesday 2nd December

• Multicore status. Queues available (63%)
  • YES: RAL T1; Brunel; Imperial; QMUL; Lancaster; Liverpool; Manchester; Glasgow; Cambridge; Oxford; RALPP; Sussex (12)
  • NO: RHUL (testing); UCL; Sheffield (testing); Durham; ECDF (testing); Birmingham; Bristol (7)

• According to our table for cloud/VMs (26%)
  • YES: RAL T1; Brunel; Imperial; Manchester; Oxford (5)
  • NO: QMUL; RHUL; UCL; Lancaster; Liverpool; Sheffield; Durham; ECDF; Glasgow; Birmingham; Bristol; Cambridge; RALPP; Sussex (14)

• GridPP DIRAC jobs successful (58%)
  • YES: Bristol; Glasgow; Lancaster; Liverpool; Manchester; Oxford; Sheffield; Brunel; IC; QMUL; RHUL (11)
  • NO: Cambridge; Durham; RALPP; RAL T1 (4) + ECDF; Sussex; UCL; Birmingham (4)

• IPv6 status
  • Allocation - 42%
  • YES: RAL T1; Brunel; IC; QMUL; Manchester; Sheffield; Cambridge; Oxford (8)
  • NO: RHUL; UCL; Lancaster; Liverpool; Durham; ECDF; Glasgow; Birmingham; Bristol; RALPP; Sussex

• Dual stack nodes - 21%
  • YES: Brunel; IC; QMUL; Oxford (4)
  • NO: RHUL; UCL; Lancaster; Glasgow; Liverpool; Manchester; Sheffield; Durham; ECDF; Birmingham; Bristol; Cambridge; RALPP; Sussex, RAL T1 (15)

Tuesday 21st October

• High loads seen in xroot by several sites: Liverpool and RALT1... and also Bristol (see Luke's TB-S email on 16/10 for questions about changes to help).

Tuesday 9th September

• Intel announced the new generation of Xeon based on Haswell.

Empty

Empty

Wednesday 15th April 2015

• Operations report
• On Wednesday 8th April a minor network change triggered a wider problem across the Tier1 network and caused significant disruption of services. The planned Castor upgrade for that day had to be abandoned. A post mortem is being generated for this incident.
• Castor was successfully upgraded to version 2.1.14-15 this morning.
• The CREAM CEs will be turned off on 5th May. (Leaving only ARC CEs).

Empty

Empty

Atlas S&C week 2-6 Feb 2015

Production

• Prodsys-2 in production since Dec 1st

• Deployment has not been transparent , many issued has been solved, the grid is filled again

• MC15 is expected to start soon, waiting for physics validations, evgen testing is underway and close to finalised.. Simulation expected to be broadly similar to MC14, no blockers expected.

Rucio

• Rucio in production since Dec 1st and is ready for LHC RUN-2. Some fields need improvements, including transfer and deletion agents, documentation and monitoring.

• Rucio dumps available.

• Dark data cleaning

• files declaration . Only Only DDM ops can issue lost files declaration for now, cloud support needs to fill a ticket.

• Webdav panda functional tests with Hammercloud are ongoing

Monitoring

• Main page

• DDM Accounting

• space

• Deletion

ASAP

• ASAP (ATLAS Site Availability Performance) in place. Every 3 months the T2s sites performing BELOW 80% are reported to the International Computing Board.

Empty

Empty

• N/A