Integration and testing of the DAQ system for the CMS Phase 1 pixel upgrade

The CMS pixel detector phase 1 upgrade in 2017 requires an upgraded data acquisition (DAQ) system to accept higher data rates. A new DAQ system has been developed based on a combination of custom and standard μTCA parts. Custom mezzanines on FC7 AMCs [1] provide a front-end driver for readout, and a front-end controller for configuration, clock and trigger. The DAQ system is undergoing a series of integration tests including readout of the pilot pixel detector already installed in CMS, checkout of the phase 1 detector during its assembly, and testing with the CMS central DAQ. This paper describes the DAQ system, integration tests and results, and an outline of the activities up to commissioning the final system at CMS in 2017.

A : The CMS pixel detector phase 1 upgrade in 2017 requires an upgraded data acquisition (DAQ) system to accept higher data rates. A new DAQ system has been developed based on a combination of custom and standard µTCA parts. Custom mezzanines on FC7 AMCs [6] provide a front-end driver for readout, and a front-end controller for configuration, clock and trigger. The DAQ system is undergoing a series of integration tests including readout of the pilot pixel detector already installed in CMS, checkout of the phase 1 detector during its assembly, and testing with the CMS central DAQ. This paper describes the DAQ system, integration tests and results, and an outline of the activities up to commissioning the final system at CMS in 2017.

K
: Data acquisition concepts; Detector control systems (detector and experiment monitoring and slow-control systems, architecture, hardware, algorithms, databases); Front-end electronics for detector readout; Particle tracking detectors

Introduction
The current pixel detector [1] consists of three barrel layers (BPIX) at radii of 4.4 cm, 7.3 cm and 10.2 cm, and two forward/backward disks (FPIX) at longitudinal positions of 34.5 cm and 46.5 cm extending radially from about 6 cm to 15 cm. The BPIX contains 48 million pixels covering a total area of 0.78 m 2 and the FPIX has 18 million channels covering an area of 0.28 m 2 . These pixelated detectors produce 3-D measurements along the paths of the charged particles with single hit resolutions between 10 -20 µm. The current pixel readout electronics was designed and optimized for the data rates and pixel occupancies expected up to the LHC design luminosity of 1 x 10 34 cm −2 s −1 with 25 ns bunch spacing. The goal of the Phase 1 upgrade is to replace the present pixel detector with one that can maintain a high tracking performance at luminosities up to 2.5 x 10 34 cm −2 s −1 . Data losses in the read out chip (ROC) and reaching the maximum throughput rate for data links of the innermost layer of the current pixel system indicate the present system will not sustain the luminosity conditions expected in Phase 1. The plan is to install and commission the phase 1 upgrade pixel detector with modified data acquisition (DAQ) and detector control systems (DCS) during an extended year-end technical stop at the end of 2016 and to have the new detector fully operational soon after.
For the CMS pixel phase 1 upgrade [2] an additional layer will be added in the forward and barrel pixel detectors, thus increasing the number of required readout links from 1120 to 1696 for the barrel and from 448 to 672 for the forward pixel system, yielding a total of 2368 readout links. The first and second layer of the barrel system will use four and two links per module, respectively, to cope with the high occupancy and data rate. In addition, the new pixel detector will feature a fully digital readout system including new backend electronics. The new digital ROCs [3] operate on a 40 MHz clock and have a 160 Mbps serial output data stream. Two streams of eight ROCs each are multiplexed in a 320 Mbps stream by the token-bit manager chip (TBM). This stream is then encoded using a four-to-five bit scheme to reduce bit-errors during transmission [4]. The output signal of the TBM is converted to an optical signal off the module and transmitted on a fiber to a Front-End Driver (FED) board in the backend electronics. In order to fit all the logic required for the decoding and deserialization of 24 input channels of a single FED card, the µTCA-based CMS Tracker Advanced-Mezzanine Card (CTA) [5], a revised version of the FC7 [6], has been selected as platform for the new digital FED. Since the µTCA standard offers several distinct advantages, such as modern FPGAs, higher speed communication, etc., over the presently installed VME-based system, for which spare-parts are limited and components are no longer available, it is planned to also upgrade the backend part of the control system of the Phase 1 pixel detector to use the CTA as Front-End Controller (FEC).

System requirements for DAQ and readout architecture
The design of the backend electronics for the Phase 1 pixel detector follows a µTCA schema. It uses a redundant, dual-star µTCA backplane to distribute clock, trigger and fast commands that are received from the central CMS Trigger and Command Distribution System (TCDS) [7] via a specifically developed module called AMC13 [8].
An overview of the system architecture including auxiliary components required to interface with the central CMS services is shown in figure 1. The pixel data from the detector front-end is sent to the FED, which decodes the incoming data stream and assembles the channel data of all 24 input channels into event fragments that are then pushed to the central DAQ. The control part of the system requires two different flavors of FECs, the Pixel FEC and the Tracker FEC, which are based on the CTA and use identical hardware. The Pixel FEC distributes clock, trigger and fast signals to the pixel modules and in addition is responsible for controlling the Digital-to-Analog-Converter (DAC) registers of the ROCs and the TBM. The Tracker FEC is used to program the registers of the auxiliary electronics and components like opto-hybrids and DC-DC converters [9] via the I 2 C interface and PIA port of a Command Control Unit (CCU) [10].
Overall, a total of 124 AMCs, with 108 being FEDs, 16 Pixel-FECs and 2 Tracker-FECs, distributed over a total of 12 crates, will be required to control and readout the barrel and forward pixel detector.

Phase I Pixel FED and FECs
The CTA shown in figure 2, which is a variant of the FC7 card, is a full-size, double-width Advanced Mezzanine card holding a Xilinx Kintex 7 FPGA [11] and offering two low-pin-count compatible (LPC) FPGA Mezzanine Card (FMC) slots. These slots allow the use of a variety of mezzanines for various applications, including compatibility with multi-Gigabit optical transceivers. The number of high-speed (10 Gbps) serial links to the FPGA available on the front-panel and backplane connector is 20 and 12, respectively. Moreover, there is a block of 4 Gb DDR3 RAM for data buffering that supports a transfer rate of 30 Gbps. The firmware can be uploaded to the FPGA via a µSD card. The application specific FMCs and firmware make a CTA into a FED or FEC.
The data from the detector modules will be transmitted to the backend via a 400 Mbps digital protocol. Since this already includes four-to-five bit encoding, the real bandwidth available for pixel   data is 320 Mbps which is sufficient to accommodate data rates expected from modules closest to the interaction point. The hardware of a Pixel FED consists of a CTA board with a Receiver-FMC (Rx-FMC). The Rx-FMC is a mezzanine having two 12-channel optical receivers that collect signals from the pixel detector, and one SFP+ 10 Gbps S-Link Express transmitter for data transmission to the central DAQ. One CTA FED can read out 24 channels and transmit output data at 10 Gbps. The receivers [12] used on the FMC are manufactured by FITEL and were specifically adapted to the 1310 nm wavelength and tested at the speeds at which the pixel-opto-hybrid (POH) transmitters operate. Figure 3 (left) shows a picture of a Rx-FMC with a small form-factor pluggable (SFP+) transceiver attached to it. The Pixel FED also needs to receive clock and trigger signals from the TCDS system and the AMC13 via the µTCA backplane, in order to read out the module data. In addition, the Pixel FED sends a trigger-throttle signal (TTS) to the AMC13 which forwards this signal to the trigger system. The operation of the CMS pixel phase 1 detector requires two kinds of Front-End Controller boards that are functionally equivalent to the present VME parts. Both devices use identical hardware but differ in the firmware running on the FPGA. The Tracker FEC offers a way to program auxiliary pixel supply electronics, via the CCU chips' I 2 C and PIA interfaces, which is completely independent from the control of the pixel modules. The distribution of clock and trigger signals via the Tracker FEC is not used for the pixel detector. Each Tracker FEC can connect a number of CCU chips in a ring-like topology via redundant optical fiber connections that carry clock and data signals. The control is done via a token-ring protocol. The digital-opto-hybrid (DOH) transceivers convert the signal from optical to electrical levels and forward it to the CCUs. The µTCA implementation of the Tracker FEC will use the CTA board and, to be compatible with the fibers and transceivers already installed in the detector, an FMC that can run at the lower speed (80 Mbps) of the control links is adapted. Figure 3 (right) shows a SFP FMC. For the CMS phase 1 pixel upgrade two µTCA Tracker FECs with two FMCs will be required to drive a total of four redundant CCU control-rings. The Pixel FEC is completely identical to the Tracker FEC in terms of the hardware. However, the underlying communication protocol, and thus firmware for the FPGA, differs. The Pixel FEC is responsible for distributing clock, trigger and fast signals to the detector front-end and for programming the DAC registers of the TBM and ROC chips on the detector modules. Commands are sent to the modules via optical fibers through a modified version of the DOH (mDOH) mounted on the supply electronics. This connection is not redundant. The clock and fast signals such as trigger, reset and calibration pulse request, that are encoded in the clock line, are decoded after the opto-electrical conversion by Tracker Phase Locked Loop (TPLL) [13] and Quartz Phase Locked Loop (QPLL)-chips and forwarded to the modules on dedicated lines. The DAC programming commands for ROC and TBM chips are carried on the data line of the control link and are routed to the front-end through a series of delay and addressing ASICs. Each module connected to a pixel-control link is identified by a unique, hardwired four-bit hub address. A total of 16 µTCA Pixel FECs will be required for the Phase 1 detector.

System tests in the laboratory
The firmware and software development is ongoing for the µTCA components. Small scale systems are used for development and testing of final detector parts, which advance development and help uncover errors and defects. There are three integration centers using µTCA backend: Fermilab, University of Zurich and CERN. At Fermilab final check out of the FPIX detector is performed before being shipped to CERN for installation in CMS. At University of Zurich [14] the focus is on testing the optical components of the service cylinders and the integration test for BPIX. At CERN emphasis is on firmware development and hardware tests. Functionality tests are also performed on detector components upon arrival at CERN.
A so called soak test facility has been set up at CERN. The rack layout is identical to the final setup. The goal is to test all production parts before they are installed in the CMS service cavern. The soak test includes firmware upload and power cycling of FED and FECs. It is a a test for infant mortality while stressing the components. Moreover, a front-end emulator board (FED tester) is designed to stress-test the FED firmware and perform measurements of optical receivers.

System tests in the CMS detector -Phase 1 pixel pilot system
In order to be best prepared for a short commissioning period and to take advantage of the long shutdown during Long Shutdown 1 (LS1), a pilot system [15] was built which consists of eight prototype pixel modules incorporating the new readout chain. The pilot system was installed in 2014 in the available space in the existing FPIX half cylinders, which host the auxiliary electronics. A hybrid solution with new daughterboards on the existing FED was used at the beginning to readout the new fully digital pixel system. The new µTCA FED system has been in use since March 2016. The motivation for installing the pilot system was to learn how the readout, control, and offline systems perform in the CMS environment of CMS. Studying the new pixel detector behavior during the 2015-2016 CMS operations provided valuable experience and enabled an early start to the modifications that are required for the DAQ.
As shown in figure 4, the pilot system half disks are integrated into the existing FPIX half cylinders and commissioned using a test stand running a standalone test software at CERN. Prototype detector calibration procedures implemented in the online software were validated after the installation in CMS. The pilot system required addition of new hardware to the pixel DAQ system at P5. The pilot system was initially controlled and read-out from dedicated VME boards placed in a dedicated pilot system VME crate. The pilot VME crate has a pixel front-end controller (FEC) motherboard with two mezzanine front-end controllers (mFECs), each connected to one of the pilot system half-disks, which are serviced by separate optical fiber ribbons. Devices on the auxiliary electronics are programmed by a separate tracker FEC with two mFECs, each connected to the pilot CCU boards in each half cylinder. One modified FED card with two daughter boards was used to receive the optical links from the two pilot system half-disks. One S-link connection, a data-link that can be used to connect front-end to readout at any stage in a dataflow environment to the central CMS DAQ, allowed readout of the pilot system FED in the usual data stream. Since March 2016 a prototype µTCA FED has been used with the pilot system. The µTCA FECs are installed in the pilot µTCA crate alongside with the µTCA FED.
The pilot system took collision data in CMS with the current pixel detector during 2015-2016 proton-proton runs both with VME and µTCA FEDs.

System tests in the future
Until now only small scale systems with low data throughput have been tested. The DAQ system will be fully qualified for the highest expected data rates before installation. A fully loaded crate (at least 8 FEDs) will be connected to the central DAQ processor of CMS sending emulated data patterns. The new Pixel detector DAQ will also be tested with all other CMS sub-systems to ensure a smooth start-up in 2017.

Conclusion
A new CMS phase 1 pixel DAQ system has been developed based on a combination of custom and standard µTCA parts. The DAQ system is undergoing a series of integration tests including readout of the pilot pixel detector already installed in CMS, checkout of the phase 1 detector during its assembly, and testing with the CMS central DAQ. It will also be tested for the high data rates expected during LHC running starting from 2017.