The development of the global feature extractor for the LHC Run-3 upgrade of the L1 calorimeter trigger system

The Global Feature Extractor (gFEX) is one of several modules in the LHC Run-3 upgrade of the Level 1 Calorimeter (L1Calo) trigger system in the ATLAS experiment. It is a single Advanced Telecommunications Computing Architecture (ATCA) module for large-area jet identification with three Xilinx Virtex UltraScale FPGAs for data processing and a system-on-chip (SoC) FPGA for control and monitoring. A pre-prototype board has been designed to verify all functionalities, which includes one Xilinx Virtex-7 FPGA, one Zynq FPGA, several MiniPODs, MicroPODs, DDR3 SDRAM and other components. The performance of the pre-prototype has been tested and evaluated. As a major challenge, the high-speed links in FPGAs are stable at 12.8 Gb/s with Bit Error Ratio (BER) <; 10-15 (no error detected). The low-latency parallel GPIO (General Purpose I/O) buses for communication between FPGAs are stable at 960 Mb/s. The peripheral components of Zynq FPGA like DDRs, UART, SPI flashes, Ethernet and so on, have also been verified. The test results of the pre-prototype board validate the gFEX technologies and architecture. Now the prototype board with three UltraScale FPGAs is on the way.


I. INTRODUCTION
he Large Hadron Collider (LHC) will undergo a series of significant upgrades during the next ten years, which allow both collision energy and luminosity increase. The ATLAS experiment will also follow the same upgrade schedule [1]. During the so-called Phase-I upgrade, the ATLAS first level trigger (Level-1) will be updated with three new components of the Feature Extractor in the calorimeter system (called L1Calo) against increasing luminosity: the Electron Feature Extractor (eFEX) [2], the Jet Feature Extractor (jFEX) [3], and the Global Feature Extractor (gFEX). The gFEX receives the entire calorimeter data in a single module in Advanced Telecommunications Computing Architecture (ATCA) shelf and thus enables the use of full-scan algorithms. The gFEX is used to select large-radius jets, typical of Lorentz-boosted objects, by means of wide-area jet algorithms refined by subject information. It is intended to identify patterns of energy deposition in the calorimeter associated with the hadronic decays of high quarks, and exotic particles in real time at the LHC crossing rate.

II. ARCHITECTURE
The gFEX is a single module with several large Field-Programmable Gate Arrays (FPGAs) for data processing and a combined FPGA & CPU System-on-Chip (Hybrid FPGA) for control and monitoring. A special feature of the gFEX is that it receives data from the entire calorimeter, enabling the identification of large-radius jets and the calculation of wholeevent observables. Each processor FPGA has 2π azimuthal (φ) coverage for a slice in pseudorapidity (η) and executes all feature identification algorithms. The processor FPGAs communicate with each other via low-latency GPIO data buses while input and output to the board are via Multi-Gigabit Transceivers (MGTs). A simplified functional representation and interface of the gFEX module is shown in Fig.1. The gFEX is a customized ATCA module based on the PICMG ® 3.0 Revision 3.0 specification [4]. The gFEX module will be placed in a sparsely populated ATCA shelf so that it can occupy two slots if needed: one for the board and one for cooling (e.g., large heat sinks), fiber routing, etc. A pre-prototype of the gFEX has been designed to verify all the functionalities of the chosen technical solutions: the power on sequence, power rails monitor, clock distributions, MGT link speed and high speed parallel GPIOs. As shown in the Fig.2, one Hybrid FPGA (ZYNQ) and one processor FPGA are included in this pre-prototype. There are also several T MiniPODs [6], MicroPODs [7], power modules and high speed parallel GPIOs on the board.

III. LABORATORY TEST RESULTS
In order to verify the functionalities of this pre-prototype and evaluate its performance, several tests have been carried out in the laboratory. According to the test results, the power on sequence control circuit as well as power rails monitoring circuit work as programmed. And the boot modes of two FPGAs have been verified: ZYNQ can boot from JTAG, SPI flash and SD card; processor FPGA can boot from JTAG and SPI flash. As ZYNQ has a processing system, it has a rich set of peripheral interfaces. All of its interface parts have been verified, including DDR3 memory, SPI flash, UART, Ethernet and so on. All the hardware technologies work as expected. As the high speed serial solution is the major challenge, the link speed of both ZYNQ and processor FPGA have been tested with the IBERT tool provided by the Xilinx Vivado. When all the 80 channel GTHs of the processor FPGA and 16 channel GTXs of the ZYNQ are enabled, all these links are stable at a rate of 12.8 Gb/s with no error bit detected (Bit Error Rate <10 -15 ).

IV. COMMISSIONING TEST RESULTS
In order to determinate the link speed to use for the links between LAr (Liquid Argon Calorimeter) and L1Calo, the LAr-L1Calo link speed test has been carried out on January 2016. The test setup is shown in Fig.3. In this test, gFEX receives data from the 48 links of LAr Digital Processing Blade (LDPB) AMC card, with FELIX (Front End LInk eXchange) used for clock distribution. In order to be similar to the real application, all of these 48 links are required to run simultaneously. The FOX (Fiber-Optical eXchange) also has been implemented in the test. Test results show that the links between gFEX and LDPB are stable at both 6.4 Gb/s and 11.2 Gb/s without any error data observed, excluding two links that have known issues at LDPB TX side. Fig.4 shows the open area of tested eye diagrams for these 48 links at 11.2 Gb/s (links of 0 and 9 have known issue at LDPB TX side). The blue line indicates the test result when both gFEX and LDPB AMC card use their local oscillator as the reference clock for high speed links. The red one is the test result with the FELIX clock distribution and one stage of FOX. The green line is the results when two stages of FOX have been configured. The high speed links from gFEX to Level-1 Topological Processor module (L1Topo) [5] also have been tested. The test results show that all of the 12 links are stable at 12.8 Gb/s, with no error observed.
As the link speed test between LDPB and eFEX also get good eye opening and BER at 11.2 Gb/s, LAr-L1Calo agrees to use 11.2 Gb/s as the baseline link speed, with option of 9.6 Gb/s and 12.8 Gb/s.

V. PROTOTYPE DESIGN
The laboratory test and link speed test results of the preprototype have verified the validity of the chosen technologies and architecture. Now a much more advanced prototype is on the way, which includes all the functionalities of gFEX with three Virtex UltraScale FPGAs used for calorimeter data processing.
As clock quality is very important for high speed links, the clock distribution should be considered deliberately. The gFEX uses the recovered clock from the FELIX link. And FELIX receives the clock information from the TTC (Timing, Trigger and Control) source. As the recovered clock is not good enough for high speed links, especially for links running at speeds above 10 Gb/s, a good clock generator should be adopted on board.
We tested several clock generator chips to evaluate their performance. The input clocks of these clock generator chips are the recovered clock in gFEX, and the frequency of their output clocks is 40 MHz. The tested phase noise of these chips is shown in Fig.5. As the test results show, clock generator of Si5345 has a better performance for jitter cleaning to improve recovered clock quality. It is used in gFEX prototype. The design of the gFEX prototype has been finished. Its layout is shown in Fig.6. The gFEX prototype is a 26 layers board with 26 MiniPODs mounted. The back drilling technology is adopted in its fabrication to reduce the influence of stubs to high speed links performance.

VI. SUMMARY
A pre-prototype board of the gFEX has been designed. All of its functionalities have been tested and verified. The parallel data buses can work well at 480 MHz in DDR mode, and the high speed serial links are stable at 12.8 Gb/s. The link speed test at CERN shows that the links from LDPB to gFEX can run properly at a rate of 11.2 Gb/s. Based on these test results of the pre-prototype, gFEX prototype has been designed with a new clock generator chip of Si5345.