Post-installation status of the ATLAS pixel detector

The ATLAS pixel detector was installed in June 2007 and was fully connected and operating at the time of this conference. An assessment is given of the state of the as-installed system in the context of the technological challenges of hybrid pixels. Comparisons with CMS are made drawing on material presented at this conference. This paper is intended as a companion to the talk slides presented at the conference and excludes the many photographs from the talk.


Introduction
The pixel detector is the innermost element of the ATLAS "Inner Detector" charged particle tracking system [1]. The pixel layout, electronics, and sensors are described in detail in reference [2]. This paper is a companion to the presentation slides shown at the conference and the photographs from the talk are not included.
The construction and assembly of the pixel detector took place between 2003 and Summer 2007, when the full instrument was lowered into the ATLAS cavern. The lowering date was approximately one year later than projected in 2002 [3]. The dominant factors in this delay were the rate of assembly and testing of bump-bonded modules and the re-manufacture of mechanical support structures for barrel modules, in response to aluminium cooling pipe failures due to corrosion. The commissioning of the detector was delayed by 2 years relative to the 2002 projection, to Spring 2008. The "extra year" delay in the commissioning relative to construction was due to accumulated delays in connection and check-out of the full Inner Detector. Because the pixel detector was the final element installed, its services could not be connected until all other sub-detector services were ready to be "sealed off". Primarily, problems with some elements of the evaporative cooling system, requiring retrofits, delayed the starting point at which connection of the pixel services could begin.

Technical challenges
Hybrid pixel technology is being used on a large scale for the first time in the LHC generation of detectors, of which ATLAS and CMS are the largest and of similar size. There are many technical challenges that had to be faced and it is interesting to compare those for which ATLAS and CMS made different technical choices. The title of each of the following sub-sections denotes a particular technical challenge, with description of technical choices, status, and discussion in the body of the section.   Table 1. Dead channels counts for ATLAS [5] and CMS [6] pixels before and after installation. (*) This is dominated by cooling circuit leaks, which were found to not prevent operation and therefore do not carry over to total lost channels.

Unproven long term reliability
Every new technology must overcome the lack of a history that can be used to predict the expected deterioration with time. The design must therefore be able to tolerate some level of unspecified degradation. The different ATLAS and CMS approaches in this respect have been partly driven by the overall detector integration design. CMS has taken advantage of faster access to the interaction region and built a pixel detector that can be rapidly inserted and removed, with services connection time of a few days [4]. While accessing the interaction region in both cases involves moving large end-cap components, in ATLAS this process is more constrained and takes longer than in CMS. Therefore, the ATLAS strategy was to build a pixel detector that can not be easily removed, and instead to address the reliability challenge with fine granularity of services and control. Granularity of services and speed of connection counter one another. Each ATLAS pixel module has independent electrical services, leading to about 25,000 electrical contacts for the pixel detector at the very crowded ends of the Inner Detector. These took about 4 weeks to connect and dress. An additional 4 weeks was needed for the 88 independent cooling circuits and 588 optical fiber ribbons. This fine granularity allows independent external control of individual modules so that single failures, whatever their nature, do not result in large channel count losses, and so that unexpected behavior can be mitigated by fine tuning from the outside. The as-installed fraction of non-operable (lost) channels in ATLAS [5] and CMS [6] is compared in table 1. In ATLAS the losses are smaller, but permanent and can only increase with time, whereas CMS has the option during normal shut-downs to remove detector elements and perform repairs. It remains to be seen what loss level will justify such repairs, because, while possible, detector manipulation is not without risk.
The dominant failure mode leading to permanently lost channels in ATLAS was sensor bias voltage open connections (each connection affecting 1 module out of 1744). Leaks in end cap cooling connections constitute the failure mode affecting the largest number of channels; however, such channels are not considered lost because the cooling system continues to perform well with the leak levels in question and therefore the affected channels remain fully operable. The 6% loss reported in the CMS end caps is dominated by a single power connection accounting for 3.1%, with 3 separate electrical contacts making up the rest. This exemplifies how the ability to quickly (dis)connect the detector implies a coarser granularity of services.

Power cost of high data rates
Processing and transmitting hits requires power. ATLAS has implemented on-chip digitization and on-detector processing of digital information. The higher power cost of this processing has been offset by a C3F8 evaporative cooling system [7]. The ATLAS system is shared between the pixel and silicon strip detectors. CMS has implemented an analogue pixel readout system, which can achieve lower power, and has successfully implemented mono-phase cooling. Nevertheless, cooling remains one of the main challenges of hybrid pixel technology, because the power load per unit active area (of order 0.5W/cm 2 ) is roughly an order of magnitude higher than mature silicon strip technology, while the requirements on mass per unit active area of the same order. Even though ATLAS and CMS have very different cooling systems, the demands of meeting the pixel power density challenge translated into aggressive cooling plant installations, and both detectors experienced difficulties with cooling plant commissioning.
In the case of ATLAS, the main problem area was the electric heaters built into the return plumbing of the evaporative cooling system. The function of these heaters is to provide a constant load on the plant (detector + heaters = constant load) such that always fluid flows to the detector and gas flows back to the cooling plant, both at room temperature. Each heater must therefore deliver up to 400W per circuit in the case of pixels, which is electrically supplied at 120V. The heater must also be able to regulate its power between maximum and minimum depending on the detector load present, by simply sensing the temperature of its exhaust gas flow. All this must be accomplished in a very restricted space-each heater is essentially a tube only slightly bigger than a passive pipe sized for the same flow. The original heater design had voltage isolation problems in the electrical feed-throughs, leading to ground shorts, and had to be refurbished with a completely different feed-through design. The process of diagnosing the source of the shorts and qualifying and manufacturing a new design was the main reason for the delay in finalizing the Inner Detector commissioning. In addition to this, the original placement of temperature sensors was inadequate for reliable regulation in the crowded environment and range of orientations of the installed heaters. During commissioning, in many cases temperature sensors intended for monitoring were used instead of the original control sensors to improve regulation.
In addition to the heaters mentioned above, other parts of the cooling plant external to the detector also failed. Most notably, 3 out of 7 compressors sustained serious internal damage and had to be re-built, which led to an additional 3 month delay in commissioning. Assuring reliable steady-state operation of the ATLAS evaporative cooling system remains an area of concern and improvement. The compressors suffered mechanical failures in magnetic drive couplings. New diagnostic tools have been added since this incident to provide early warning of heating of internal compressor elements, enabling a safe shut-down before mechanical damage happens.

Long distance data transmission
The large size of the LHC detectors means that pixel data must travel 50-100m to the data acquisition system. Optical communication offers the needed bandwidth, range, and ground isolation. Therefore, both ATLAS and CMS have implemented optical systems, but CMS has an analogue system where the light level encodes value information, whereas in ATLAS the optical links are

JINST 4 P03021
purely digital. Both experiments developed full custom optical links to meet their needs, and both have encountered unexpected difficulties in operating them reliably.
A notable common problem has been the temperature sensitivity of the on-detector optical transmitters. Variation of light output with temperature is of obvious importance to CMS due to their analogue readout, but even for ATLAS it turned out to be critical, because 8-element vertical cavity surface emitting laser (VCSEL) arrays are used for transmission. The array bias current is adjustable in common, but not independently for each channel (each array element serves one module). The temperature variation affects the uniformity of the transmitter arrays, which is important due to common bias current. This behaviour was discovered during system tests before detector installation [2] and, as a consequence, resistive heaters with independent control were added to the on-detector optical circuit boards. These heaters enable temperature regulation by setting the coolant evaporation temperature below the desired value and actively controlling the heaters to reach and maintain the operating temperature. The typical operating temperature is 20 • C (by contrast the ambient in the detector volume is near 0 • C). Even with this added environmental control, achieving error-free operation of the optical links has dominated the calibration effort and requires frequent checks to detect channels that must be re-tuned [8].
Independently from the above temperature effects, ATLAS has observed a significant number failures in the off-detector optical transmitters (see figure 1), which send commands and configuration to the modules. These use the same VCSEL technology as the on-detector transmitters, yet no comparable failure rate has been observed on-detector. As of this writing, there are 4 channels with failures that are likely to be in on-detector VCSELs, but it is not possible to confirm their exact nature (because the detector is inaccessible). The off-detector transmitters are fully accessible and can be replaced as they fail, whereas the on-detector transmitters are in the detector volume and can not be reached except by fully removing the detector. Study of failed components suggests that the VCSEL arrays may have suffered electro-static discharge damage at some point during assembly. Illustration 1 shows number of failures vs. time. Each failure affects one module out of 1744 total in the detector. The failed off-detector parts have all been replaced at the time of this writing, so they do not represent a long term loss of functionality.

Recovery path for radiation damage
The innermost layers of ATLAS and CMS are expected to receive a 1 MeV neutron equivalent dose that could exceed 1 × 10 15 cm −2 during the LHC program. At the time the detectors were designed, it was known that this may result in severe degradation of the hybrid pixel technology used, but no technology with demonstrated higher tolerance was available. Therefore, the possibility of refurbishment of inner layers was foreseen. For CMS, the relatively fast removal and installation enables a true refurbishment program, and this is the expected recovery path. For ATLAS the situation is more complicated. While the original pixel detector concept anticipated an independent inner layer, with dedicated service connections [9], it was not possible to implement this arrangement in the final design. In ATLAS, the pixel detector services and the central beam pipe section are coupled, forming a "packaged unit" that can only be integrated or dismantled on the surface. Based on measured integration times for the installed pixel package, it has been determined that a traditional refurbishment, where the detector is removed to the surface, refurbished, and re-installed, is not possible during any reasonably anticipated accelerator shut-down. The recovery path for ATLAS now focuses on the insertion of a new barrel layer inside the present detector, without direct actions on the detector itself. The procedure for this would be to remove the beam pipe in-situ (without moving the pixel package out of position) and then insert a new beam pipe with pre-assembled pixel layer already on it. The concept is similar to the beam pipe layers of the CDF [10] and D0 [11] experiments. This requires a smaller outer radius beam pipe than presently used, which may lead to a reduction of the inner radius as well, but could alternatively be achieved entirely by using more efficient thermal insulation outside the vacuum chamber. The magnitude of the available space can be appreciated from the minimum radius of the inner layer of the present detector (46 mm), compared to the inner radius of the present vacuum chamber of 29 mm.
A new readout integrated circuit and a new module design are required to produce an insertable layer with full acceptance in the available radial envelope as well as increased hit rate capability. Such an integrated circuit is under development [12]. Additional challenges include the tooling for safe removal and insertion of the beam pipe, and the design for routing and support of the services for the new layer to fit inside the present detector.

Conclusion
The ATLAS pixel detector has been deployed for over 1 year at the time of this conference, never to be accessed again until end-of-life removal, and is ready for collisions. The reliability challenge for this inaccessible detector has been addressed with high granularity of services and control. The "as installed" good channel count is excellent, exceeding the design goal of 97%. The CMS detector has addressed the reliability challenge in a different way, by making the detector accessible in a relatively short down-time, which implies a coarser granularity of services.

JINST 4 P03021
The main problem areas for ATLAS pixel operation are cooling and optical links, and reliability of high and low voltage connections. Interestingly the same areas have been problematic in the CMS detector. While aggressive cooling plant designs were needed to meet the pixel power density challenge and some of the problems can be attributed to this, the optical readout systems perhaps could have been more conservative as this is not an area of great expertise within the particle physics community. Also the importance of full system functional testing is underscored by many of problems pointed out in this note.
Both ATLAS and CMS have plans to recover performance after heavy radiation damage. The ATLAS approach involves the construction of a new barrel layer to fit inside the present detector, mounted on a new replacement beam pipe. Development of a readout integrated circuit for this proposed insertable layer is under way.