Subscribe by RSSTECHNICAL COORDINATION
Operational ExperienceSince the closure of the detector in February, the technical operation of CMS has been quite smooth and reliable. Some minor interventions in UXC were required to cure failures of power supplies, fans, readout boards and rack cooling connections, but all these failures were repaired in scheduled technical stops or parasitically during access dedicated to fixing LHC technical problems. The only occasion when CMS had to request an access between fills was to search for the source of an alarm from the leak-detection cables mounted in the DT racks. After a few minutes of diagnostic search, a leaking air-purge was found. Replacement was complete within 2 hours. This incident demonstrated once more the value of these leak detection cables; the system will be further extended (during the end of year technical stop) to cover more racks in UXC and the floor beneath the detector.
The magnet has also been operating reliably and reacted correctly to the 14s power cut on 29 May (see below). In order to minimize the mechanical ageing of the coil due to the stresses of field-cycling, the magnet will be kept at full field as much as possible. It will be switched off only for essential maintenance of its cryogenic, power or cooling systems or for work in UXC which is incompatible with the stray field. All such work will be grouped together as much as possible. There is now considerable accumulated experience of working in the UXC with the magnetic field on. We have recognised the need to expand our kit of non-magnetic tools, but otherwise most of the maintenance and repair, such as topping up the fluorocarbon cooling plants, can be done with the magnet at 3.8T.
Accesses to UXC are now all being organized through a Web interface to the Equipment Management Data-base (EMD). Each intervention has to be declared, approved and monitored as a work package. The system also monitors the material flow in and out of UXC through the radioprotection buffer zone. Individual sub-projects are responsible for requesting work packages, which are then reviewed and authorized by TC. The system is reliable and easy to use and its use is now mandatory. We would like to remind the collaboration that though granting an access is a well understood procedure, with a minimized administrative overhead, it still requires quite substantial supervision, both from CMS and LHC. We therefore ask subsystem teams to stay aware of the access status and to think ahead. This will help avoid last-minute requests, in particular those arriving after the cavern has already been closed again after a scheduled access, which, except in emergency cases, are likely to be refused!
In preparation for operating CMS with a central shift crew only, the alarm- and action-matrices of all subdetectors and the central safety and control systems have been reviewed during the year. Recently, after several iterations and an extended test phase under parallel central and local supervision, a final review was held to authorise detector components one by one for unattended operation under supervision by the central CMS technical shift. For EB, EE, ES, HB, HE, HF, DT, CSC and RPC the permission for central operation was given straight away. Pixel and Tracker soon followed, after implementing a lock-off mechanism, requested by TC, which prevents central DCS from switching them back on after they have been switched off (by an alarm or an expert), unless expert permission is given. ZDC and CASTOR are the only components which have not yet provided sufficient information to be operated centrally without an expert being present in the control room.
Despite a sound and maturing monitoring system, performing a twice-daily safety tour, including the accessible underground installations, is still highly desirable. The tour has been developed over the last couple of months and has proven to be justified, as several minor problems have been uncovered. However, it is often difficult for the central shift crew to find the time for the tour. It has therefore been decided to train our technical staff at Point 5 to perform these tours. The safety tour will be one duty of a CMS technical piquet service that will be available to help with technical incidents and searches of UXC in case the “patrol state” is lost. Besides helping our shift crew this measure also helps to maintain a close connection between the experiment and our technical staff, outside technical stops and shutdowns.
On Friday May 29 at 23:30, the CMS safety system and incident recovery procedures underwent a realistic test, when a circuit breaker flash-over caused an LHC-wide power cut of around 14s. Cooling and the cryogenic system consequently went down as well. It is encouraging to note that all safety systems worked as expected and the experiment was completely safe throughout the incident. The magnet went into slow discharge using Helium from the 6000l local dewar, as foreseen in such a case. The inertion of the tracker volume kept running normally; if needed it has several backup systems. The diesel generator at point 5 kicked in correctly and provided a minimum electrical power to monitor the situation and run safety systems such as lifts. The cold box was re-engaged after around 3 hours, so that Helium liquefaction re-started. By that time it was clear that LHC would be down for at least 24 hours, and that 9 hours would be needed to re-liquefy sufficient Helium reserve for full-field operation. As a consequence it was decided to re-establish services, then postpone restart of the experiment until the next morning. By the 4½ hour mark, all the cooling and HVAC was running again. It took about 6 hours to restart the experiment on Saturday 30 May and by 17:00 CMS was running smoothly again. It is worth remarking that this was achieved without entering UXC. With there being no prospect of physics beam before the Technical Stop of 31 May, the magnet re-start was postponed until the 3rd and last day of this stop, when a ramp to full field was achieved in 4½ hours.
Analysis of the incident nevertheless showed some room for improvement. Rack control was lost due to inadvertent switch off of a DCS PC, the cause being traced to PC racks with computers on UPS but fans not. The programmed fast discharge at the end of the magnet slow ramp-down may not be the fastest path to recovery when the cold box is already re-engaged and replenishment of the 6000l reserve Dewar is the priority.
Technical stops
The revised machine operating plan for 2010-11 currently foresees 4-day technical stops every 6 weeks, with a 1 day midweek cryo regeneration stop in between. In line with this, Technical Coordination and EAM must therefore have planning for these opportunities under constant development and update. In addition, contingency planning for a 1-week or 1-month LHC-fault-provoked stop is also prudent.
Latest LHC machine thinking envisages a 9-week extended tech stop starting around 6 December 2010. A detailed draft planning for this period already exists, with possible activities including: fixing the displaced –z end alignment ring, installing TOTEM T1, installing the UXC public address system, consolidating the Nitrogen dewars prior to SX5 OSC work, installing the PM54 PAD and shaft plug, trial installation of the ZDC crane, upgrade of the CASTOR magnetic shielding, and extension of the leak detection system. In this scenario, TOTEM activity will most likely be the critical path. T1 installation procedures and planning will be reviewed in an EDR before the end of June and a decision on the risks/benefits of proceeding should be taken by September.
Facilities and services supporting operation and maintenance
Now that the operation phase of CMS is well under way some effort can be redirected to infrastructure and other common projects at point 5 and elsewhere. The goal is to identify and prioritise necessary consolidation work, improvements required for higher luminosity, and changes or facilities that will make the operation of CMS easier and more efficient.
One project already launched is the refurbishment of parts of the SX5 assembly hall to fit the needs of subsystems for a local operations and maintenance support centre (OSC), a project whose initial phase (mostly Civil Engineering) is being led by our Point 5 EAM team. The immediate priority is to convert the SHL building into a clean room facility for the Pixel and BRM communities. This building was used to host the CMS cold box when the magnet was still on the surface. The work implies structural changes to the building and the installation of new services to support the clean room operation. Studies are well advanced and the first Civil Engineering jobs should take place before the summer. Almost as urgent is the creation of a large access-controlled zone within SX5 where activated materials can be stored or worked on; this zone will include a small workshop. Next on the high priority list is addressing the lack of office space at point 5 by constructing a new building. The SL53 existed on paper from an early stage in the design of the point 5 layout, but was never built. The plan is now to share the future available space between offices and a visitor center as well as recreational facilities like coffee or kitchen areas. The building permit is currently being reviewed by the French authorities.
Meanwhile, in Building 904, work has already started to consolidate the Electronic and Electrical Integration Centre, one of the main goals being to allow sub-detector groups and the level-1 trigger to test and validate revised versions of the readout firmware and software, while the systems at point 5 are in production. Racks have been installed for the DAQ slice (capable of reading 16 S-links at 100kHz) and work is underway to install the associated DSS, power and cooling infrastructure, plus a DCS system.
For the coming shut-downs, CMS has to be prepared to shield parts of the detector which may become radioactive. Activation is a progressive process becoming more and more severe as the average luminosity increases, but decreasing (initially rapidly) depending on the “cooling time” since high luminosity p-p operation. A working group has been set up to design shielding for the EE/ES region, the tracker bulk-head, the pump station at 13.7m, the flange at 10.6m and bellows around 3.5m and 16.3m (all distances from the ip on either end). In addition, beam pipe protection will be designed for all foreseen opening scenarios with the possibility to later shielding later. PSL, Wisconsin is leading the design. The aim is to have an initial shielding system ready for the 2012 shutdown. In parts, the design will be modular, so that material can be added only according to need, with the installation procedure being exercised using the light-weight versions initially necessary.
A second working group is evaluating how to improve the forward region around HF. The present procedures for opening and closing of HF, TOTEM and CASTOR, for engaging and disengaging the beam pipe support at 13.5m, and for beampipe load transfer prior to endcap opening, are not compatible with the ALARA principle once the area becomes significantly activated. In the same region, despite several attempts to stabilize the HF tower against movements in the magnetic field, it is observed that initial ramp-up of the magnet after re-assembling the HF-tower leads to unpredictable and potentially risky movements of the entire forward region. Fortunately, practical experience shows that subsequent ramps are predictable and reproducible. To better comply with ALARA and to avoid the initial movements, a working group for revision of the forward region has been started, led by the CERN-based infrastructure team. It is expected that the necessary changes will be realized progressively, with completion during the shutdown 2015 or 2016.
Finally, CMS is cooperating closely with the EAM-led effort to help tackle the challenges of the relocation of radiation intolerant LHC equipment from the UJ56 to other underground areas at point 5, including the CMS service cavern. This ambitious project, called R2E (Radiation To Electronics), aims at providing solutions for relocating critical electronic and electrical components which are predicted to fail (anytime after 2013) once luminosity and the associated radiation level increase sufficiently. As experience from CNGS shows, the disruption to beam operation induced by single event upset in machine electronics can be devastating. The goal is to make the best of the 2012 shutdown and possibly relocate all susceptible equipment to protected areas, some of which could be in the USC55 (mostly in S4). CMS has to be sure that there is no detrimental effect on experiment operation (eg from poor separation of grounding) and that sufficient space remains in USC for all foreseeable CMS activities (eg parallel operation of upgraded readout).
Upgrade Preparations
The list of Technical projects related to upgrade is already impressive. Some are relatively clear-cut, such as the increased SCX cooling capacity needed to accommodate the luminosity-driven HLT farm expansion. Some of the organisational structures needed require thought. An example is the evolution of the Engineering Integration task, for which a consultative review was held last week.
The proposed pattern of LHC running, with two long shutdowns in 2012 and 2015, but otherwise only annual technical stops, constrains the ability to exploit the rapid-opening design of CMS to execute changes in a progressive manner. Thus the first and most crucial project is to produce a realistic schedule of activities. The shutdown of 2015 (LINAC4 + collimation) is already packed with proposed tasks, notably the replacement of HCAL photo-transducers and front-ends and the insertion of the 4-layer, low-mass, pixel tracker. A TC task-force on the central beampipe diameter has recently concluded that there is insufficient time to specify a lower diameter beampipe for installation in 2012. This is basically because more understanding is needed about how to position the pipe axis on the actual beamline. Unfortunately, the beampipe upgrade will therefore also have to be executed in 2015, which then becomes a packed shutdown with logistics designed to maximise the time CMS is fully open. Any superimposed maintenance activities are also most like to re-inforce this fully open layout. The consequences for the shutdown of 2012 are profound. It is becoming clear that this is the only opportunity to complete the bulk of the forward muon upgrade. In preparation for this, the CERN-based engineering team recently completed the design of the YE4 shielding disk, in close collaboration with an engineer from the chosen manufacturers in Pakistan. Procurement must start soon. Similarly a task force is studying the integration of the forward muon upgrade, in preparation for an EDR in a few months time.
The timely execution of this upgrade depends primarily on releasing resources for CSC material procurement and manufacture, but it is also highly dependent on completion of the production facilities in building 904. This project, again being led by our EAM team for the Civil Engineering phase, is ready to be launched once central CERN funding is released. It will start with consolidation of the floor and roof, which will then open the way for production equipment to be installed, while gas, air conditioning and cooling installations are completed. By early 2011, CSC and RPC production should be possible, while the massive programme of consolidating the shell of the building continues outside and in parallel. Once this work is finished in June 2011, it will be possible to achieve the temperature and humidity control needed for reliable RPC testing.
by A. Ball, W. Zeuner and Martin Gastal.