Data-acquisition system developments for ATLAS pixel QA and QC test toward High-Luminosity LHC

Abstract In preparation for the High-luminosity LHC upgrade, the whole ATLAS inner tracker will be replaced by a new silicon detector tracker. The innermost region will be covered by silicon pixel detectors as a high density of produced particles is expected. To operate in such an environment, high-resolution sensors and a high-speed readout system are required. At the moment, the front-end readout chip prototype RD53A and a data acquisition system (the YARR system) based on a commercial FPGA board and dedicated software for quality assurance and quality control test have been developed. Due to the high density of sensor channels, the output speed from RD53A is at maximum 1.28 Gbps per line, sixteen times faster than the readout front-end chip currently used in the ATLAS pixel system. The data acquisition system needs to establish communication with 1.28 Gbps speed during the quality control tests, to validate the data acquisition path, and to optimize the procedure for data taking at high speed. From the quality control perspective this optimization allows to test large numbers of modules simultaneously exploiting the data acquisition speed reducing the time needed for the tests. Another challenging point is the novel concept of readout structure planned for the operation after installation. In High-luminosity LHC, large parts of the ATLAS data acquisition system infrastructure are going to be shared among all sub-detectors, using FELIX systems, while current ATLAS data acquisition systems are dedicated for each sub-detector. This means that all processes done in the present data acquisition system hardware need to be overhauled into software running on the new data acquisition system. To minimize the differences between the data acquisition system for operation and quality control test, we introduced the prototype FELIX system into the data acquisition path of the YARR system. In this proceeding, an established data acquisition structure for quality assurance and quality control test of the new pixel detector is introduced, and results from basic quality control tests of the pixel detector with the new readout chip are presented.


Introduction
The Large Hadron Collider (LHC) at CERN is the world's highest energy particle accelerator. To collect significantly large amount of data during the next decade after the LHC project, the High Luminosity LHC (HL-LHC) project is planned. At the HL-LHC, the instantaneous luminosity is planned to be increased by a factor of 5 to 7.5 with respect to the LHC design, and accordingly pile-up and radiation damage are expected to be increased significantly. The detectors used for LHC, including ATLAS [1], need to be upgraded to operate in such extreme environments. As part of the upgrade of the ATLAS detector, the current inner detector is going to be replaced by new silicon based inner trackers (ITk) consisting of pixel and strip detectors.
While the current pixel system consists of about 2000 modules, the ITk pixel system will be composed of roughly 9200 modules. Those newly designed modules need to pass Quality Assurance (QA) test and Quality Control (QC) test. At the moment, a data-acquisition (DAQ) system, so called YARR (Yet Another Rapid Readout), for QA/QC test of the prototype of front-end readout chip (RD53A) has been developed based on the PCI Express (PCIe) board. Those are introduced in Section 1.1 and Section 1.2.
Although the YARR system is available as a DAQ system for QA/QC test of the new modules, there is a significantly different concept of DAQ system for the operation of the experiment itself. In HL-LHC, large parts of the ATLAS DAQ system infrastructure for operation will be shared among all sub-detectors. In particular, the ReadOut Drivers (ROD) will be replaced by the FELIX (Front-End LInk eXchange) system (cf. Section 1.3).
To optimize the DAQ system for operation in early stages, it is ideal to minimize the differences between the DAQ system for operation and QA/QC tests. In this proceeding, a prototype FELIX system is introduced into the DAQ chain of the YARR system in Section 2, and results from basic QA/QC tests of the pixel detector are shown in Section 3.

RD53A
RD53A is the prototype of the readout chip [2] produced with 65 nm technology. The size of the chip is 2 × 2 cm 2 , and the readout pixel size is 50 × 50 µm 2 , with a total of 76800 (192 × 400) readout pixels. To investigate the best architecture of the analog part of the readout chip, RD53A has three blocks of pixels with different types of front-end circuits. RD53A has three programmable output lanes up to 1.28 Gbps, making it 16 times faster than the readout chip currently used in the ATLAS pixel system, with Aurora 64/66 protocol, while the chip is controlled with 160 Mbps single differential serial inputs. Figure 1 shows a single chip card with RD53A.

YARR
YARR is a readout system for several types of readout chips [3]. It can run on commercial FPGA boards such as XpressK7. The key concept of this system is to implement in software all the functionalities that are implemented in firmware in the current DAQ system. The FPGA works simply as an interface to the readout chip. Thanks to this concept, by switching software interface, the system is able to communicate not only with RD53A but also with the readout chip of the current pixel detector (FEI4B) and with the readout chip for ITk Strips (ABC/HCC STAR). An additional advantage is the small latency of the readout chain because of the simple structure of FPGA.
The communication of YARR with RD53A is constructed as follows. The display port from RD53A is connected to the FPGA board via an FPGA Mezzanine Card (FMC) called Ohio card. The FPGA board is mounted on a PC where the software runs, and communicates with the software via PCIe. Basic functionalities for QA/QC tests of RD53A have been developed with the YARR system.

FELIX
FELIX, the infrastructure for the ATLAS DAQ system, will start to be used for readout of some subdetectors already in Run3 (2021) and is expected to be employed by all sub-detectors for HL-LHC operation [4]. In the current ATLAS DAQ system, all sub-detectors have a dedicated DAQ chain, which processes data with protocols that are specific to each subdetector. Those dedicated DAQ chains process data with different protocols depending on the specific subdetectors. It is relatively simple to build this type of DAQ chains, but every dedicated DAQ chain requires individual maintenance. In contrast to the current AT-LAS DAQ system, the key concept of FELIX is to share the same readout electronic system between all subdetectors.
For Run 3, FELIX is implemented on FLX712 [4] which is an ATLAS custom board based on a Xilinx Kintex UltraScale FPGA ( Figure 2). It has 48 optical links to communicate with detectors, and 16 PCIe lanes to communicate with the host PC. Uplink of this system lies a 4.6 Gbps GigaBit Transceiver (GBT) protocol [5], and downlink there is either a GBT mode protocol or a custom protocol for Felix, called full-mode protocol. For development of the readout chain, a scaled down system is also available. It is implemented on VC709 which is a Xilinx evaluation board (Figure 3). It has 4 optical links to communicate with detectors, and 8 PCIe lanes to communicate with the host PC. The functionalities of the two systems are the same except for the number of communication lines. The readout chain described here has been constructed with VC709.

Readout chain
As mentioned in Section 1, it is ideal to minimize the differences between the DAQ systems for operation and QA/QC test. Therefore, FELIX boards are introduced in the YARR system as an additional interface. Since input and output protocols are different between RD53A and FELIX, those cannot communicate directly. In this section, two complementary approaches to establish communication are introduced.
One approach is to use a board called πLUP (PIxel detector high Luminosity Upgrade card) [6], which is shown in Figure 4. This board is equipped with a Xilinx Kintex-7 and Zynq-7 FPGA. It is connected to RD53A via a Low Pin Count (LPC) FMC connector using the Ohio card, as used in the YARR setup. Communication to FELIX is established via Small Formfactor Pluggable (SFP) connectors by optical fiber. The πLUP board works as protocol converter between FE-LIX and RD53A. The commands generated by software are transmitted to FELIX and are converted to optical signals with a GBT protocol. Received commands on πLUP are decoded from the GBT protocol and reconverted into a differential serial signal. The reconverted command is sent to RD53A via a display port. For downlink, the output from RD53A is configured to 160 Mbps with 4 lines in this setup. The output is transmitted with the Aurora protocol from RD53A to πLUP, and is decoded on πLUP. The decoded output is converted to full-mode protocol and transmitted via optical fiber to the FELIX system.
This setup allows to use the FELIX system without any necessary modification in the firmware, since all necessary functionalities for protocol conversion are on πLUP. The higher speed mode of downlink from πLUP (full-mode) is supported. However, for downlink from RD53A, the speed is determined by 160 Mbps in 4 lines, which is smaller than the maximum possible speed.
The other approach is to use a Versatile Link Demo Board (VLDB) [7] and Interface Card (IC), shown in Figure 5 and Figure 6 respectively. The communication structure is shown in Figure 7. Commands from FELIX are transmitted via optical fibers, which are distributed to ICs via HDMI, and then are propagated to RD53A via a display port. Downlink, the output from RD53A is configured to 1.28 Gbps with 1 line in this setup. The outputs are converted from electrical to optical signals without logical conversion on ICs, and are propagated to FELIX. All necessary logical conversion done on πLUP is performed in the FELIX firmware interface. This setup allows to test several RD53As simultaneously with the highest downlink speed from RD53A. However, downlink of FELIX, the GBT mode is used, with a lower speed. In addition, the FELIX firmware needs to be modified to implement logical conversion.
In both setups, the only modified part in the YARR software is the interface with FELIX. Therefore, all  functionalities of YARR developed for QA/QC tests of RD53A are available with the setups. In the following section, preliminary results obtained with those two complementary approaches are shown.

Results
The communication with RD53A is successful with both setups. With a digital scan, which tests the digital circuits using 100 test pulses, 100% response is received, therefore stable communication is confirmed.
In a threshold scan, a variable charge configured by a register named Vcal is injected to each pixel configured with fixed thresholds. The number of times the single pixel is above threshold as a function of the injected charge is shown in the s-curve (Figure 8). In following tests, only one of the three front-end circuits, the differential front-end which is the selected type of front-end circuits for the ATLAS pixel detector in HL-LHC, is activated. From the s-curve, a threshold value is defined from the injection charge with which a 50% response rate is obtained, and it is converted to electron charges for each pixel. Figure 9 shows the distribution of threshold before the threshold scan: it has typically a wide range before tuning.
The threshold configuration is tuned so that all pixels have the same threshold. The threshold tuning is performed by using the occupancy distribution, which is shown in Figure 10, with various threshold configu- rations and with an injection charge fixed to the target value. The threshold configuration is tuned to have an occupancy distribution of 50% for the fixed target injection charge. Since the threshold value is defined by the injection charge with 50% of occupancy, after a successful threshold tune, the threshold distribution converges around the target value. Figure 11 shows an example of a threshold distribution after threshold tuning with 1000 electron charges as the target value. Although few pixels have a larger threshold because of failure in fitting the s-curve, most pixels have a threshold close to the target value.

Conclusion
Two setups for QA/QC test of RD53A with FELIX are introduced in this proceeding: one is without modification and in full-mode, the other one is able to be