The ATLAS Level-1 Topological Processor: from design to routine usage in Run-2

During Run-2 the LHC is providing proton–proton collisions to the ATLAS experiment with high luminosity (up to 2.1•1034 cm−2s−1), placing stringent operational and physical requirements on the ATLAS trigger system in order to reduce the 40 MHz collision rate to a manageable event storage rate of ∼1 kHz, while not rejecting interesting physics events. The first level (Level-1) trigger is the first rate-reducing step in the ATLAS trigger system with a maximum output rate of 100 kHz and decision latency of less than 2.5 µs. An important role is played by its newly commissioned component: the L1 Topological Processor (L1Topo). This innovative system consists of two blades designed in AdvancedTCA form factor, including four individual state-of-the-art processors, and providing high input bandwidth and low latency data processing. Up to 128 topological trigger algorithms can be implemented to select interesting events by applying kinematic and angular requirements on electromagnetic clusters, jets, muons and total energy. This results in a significantly improved background event rejection and improved acceptance of physics signal events, despite the increasing luminosity. This is becoming more and more important for analyses making use of low pT objects, like the Heavy Flavour and Higgs physics programme. An overview of the L1Topo architecture, simulation and performance results during Run-2 is presented.


I. INTRODUCTION
ATLAS [1] is one of the main experiments at the Large Hadron Collider (LHC) [2]. The LHC provides proton-proton collisions at a rate of 40 MHz. The ATLAS experiment uses a two-level trigger system, shown in Fig. 1, to select interesting events to record and analyse offline. The first level (Level-1) is a hardware-based trigger which uses a subset of the detector information to reduce the rate of 40 MHz to less than 100 kHz, mostly due to the detector readout capabilities. This is followed by a software-based high level trigger (HLT) which reduces the event rate to around 1 kHz on average.
During the LHC shutdown after Run-1 the Level-1 trigger system was upgraded including hardware, firmware and software updates to cope with the higher luminosity (∼2·10 34 cm −2 s −1 ) expected in Run-2. In particular, new electronics modules were introduced in the real-time data processing path: the Level-1 Topological Processor (L1Topo).

II. THE L1TOPO CONCEPT
The Level-1 trigger [4] is a fixed latency, pipelined, synchronous system. It consists of a calorimeter trigger (L1Calo), a muon trigger (L1Muon) and a Central Trigger Processor (CTP). The L1Calo and L1Muon triggers are based on multiplicity of clusters and muons over an energy threshold, respectively. Using the Run-1 system, and to cope with the increase of the trigger rates in Run-2 with the luminosity, it would be necessary to raise the trigger energy thresholds and prescales resulting in an inevitable loss of signal acceptance. With the inclusion of L1Topo, for the first time at this stage, it is possible to transfer and analyse the angular and energy information of each trigger object in the event in one system and compute topological angular and kinematic selections providing a largely improved background rejection with minimal or no signal loss. Beyond the L1Topo hardware, the installation of new modules, the CMX and the MUCTPI2Topo, was required to pre-sort and forward the input data to L1Topo. Different categories of topological selections have been implemented in L1Topo according to the needs of the physics analyses and to help the commissioning of new systems. This includes angular selections like requirements on the azimuthal angle (∆φ), pseudorapidity (∆η), radial separation (∆R), box acceptance cuts, and energy threshold of objects inside a cone; different physics mass selections like invariant mass, transverse mass and effective mass; and event selections like event hardness (sum of all jets transverse momenta in the event) and corrections to the missing transverse momentum (E miss T ) via look-up tables. Furthermore, L1Topo offers the flexibility to use central and/or forward objects, to combine calorimeter and muon information for exotic signatures such Each processor FPGA receives different types of Trigger Objects (TOBs), e.g. 4-vectors of muons, electrons, taus and jets, from the calorimeter and muon triggers through 80 multi-gigabit receivers running at a link speed of 6.4 Gb/s. Therefore, a total data rate of about 1 Tb/s with a latency budget of 150 ns is being processed by a single module. as γ-µ and τ -µ, and even to have requirements on trigger objects coming from different bunch crossings as in the case of a long-lived particle whose products may decay in the next bunch crossing.

III. THE L1TOPO HARDWARE
The L1Topo system consists of a single processor crate equipped with two identical AdvancedTCA-compliant modules. A picture of one L1Topo module before fibre assembly is show in Fig. 2. Each module has two processor FPGAs (Xilinx Virtex7) to process the algorithms and one controller FPGA (Kintex 7) responsible for the readout and all communication logic to interface to the external systems. The input data (∼1 Tb/s) are received via optical fibres with a speed of 6.4 Gb/s per line to the back-plane and the front-panel from the L1Calo and the L1Muon systems, respectively. These are transformed into electrical signals and then directed into the processor FPGAs, where they are deserialised in multi-gigabit transceivers. Both processors are supplied with the same data so that they can operate on their input independently and in parallel. Up to 128 algorithms (32 per FPGA) can be executed on the real-time data path. The output consists of a decision bit and an overflow bit for each algorithm, indicating whether the event fulfilled the specific selection, and are transmitted to the CTP via electrical cables (an option via optical fibres is also available).
The latency of the L1Topo system is of ∼ 200 ns, 75 ns of which are dedicated to the computation of the topological algorithms. The total number of trigger objects with an energy resolution of ∼1 GeV and angular resolution of ∼1 rad that can be received in L1Topo is: 120 electrons/photons, 120 taus, 64 jets, 32 muons, 1 missing transverse momentum, resulting in a large amount of combinatorics inside the topological algorithms. To reduce the resource usage, in 50 ns, lists of 6 sorted trigger objects and 10 selected trigger objects above an energy threshold are created. Where possible, algorithms using the same input list are implemented on the same FPGA. Then, in the remaining 25 ns, the algorithm selections are evaluated on the reduced lists. Only a few algorithms are evaluated on the full not-reduced inputs. By the end of 2018, 113 topological algorithms have been implemented in VHDL and validated. The resource usage per FPGA is between 50% and 70%.

IV. VALIDATION
Validating the L1Topo hardware behaviour and the algorithm decision is a fundamental task to be able to use the L1Topo triggers for the ATLAS physics program. The validation is done on different levels. To ensure a correct behaviour of the hardware and all communication logic, sanity tests are performed first in laboratory conditions on a spare L1Topo module and then in ATLAS with the complete L1Topo two-module system.
The timing of the arrival of the decision outputs is also checked to ensure they are all well aligned with the triggered events.
For the validation of the algorithms, the firmware is first simulated in VHDL where basic checks are performed standalone examining the algorithm decisions for well-defined input data. In general, this allows any large problem in the implementation of the algorithmic logic to be found. Then, a complete bitwise simulation of all L1Topo algorithm decisions is compared with the hardware decisions for real events. Figure 3 shows the relative difference between the hardware and the simulated trigger decisions for a variety of topological selections, proving a very good understanding of the topological algorithms. These plots are also produced in real time during ATLAS data taking to ensure the integrity of the recorded L1Topo triggers data.

V. PHYSICS PERFORMANCE
The topological triggers offer significantly improved background event rejection and improved acceptance of physics signal events despite the increase in the luminosity. Hence, these triggers have become more and more important for highrate demanding analyses making use of low p T objects. One example is the case of the H → τ had τ had analysis [5], where the non-topological selection requires two isolated tau trigger objects of relative low transverse momentum of 20 GeV and 12 GeV. An additional requirement on the angle between the

2⌧ triggers and L1Topo commissioning
Among the tau triggers, the 2⌧ trigger is of highest priority due to its importance for the measurement of the coupling of the Higgs boson to fermions. To ensure high signal acceptance, it is essential that both ⌧ had-vis p T thresholds are set as low as possible and this calls for alternative strategies to reduce the L1 trigger rate. The L1 rate reduction of the 2⌧ trigger has been particularly critical in 2016, when the peak instantaneous luminosity exceeded the LHC design value of 10 34 cm 2 s 1 by almost 40%. Even higher luminosities are expected in 2017.
Since at L1 only calorimeter information is available, the ability to reduce the L1 rate without increasing the ⌧ had-vis p T threshold is only by means of isolation requirements on each ⌧ had-vis candidate and event-wise topological requirements. Figures 3(a) and 3(b) show the impacts of such selections on the L1 trigger rates and trigger cross sections of the 2⌧ trigger, respectively. At 1.2 ⇥ 10 34 cm 2 s 1 , for a 2⌧ trigger with L1 ⌧ had-vis p T thresholds of 20 and 12 GeV, the L1 isolation (Section 3.1) reduces the rate from 54 to 23 kHz. To further reduce the rate, the primary 2⌧ trigger used throughout 2016 requires the presence of an additional jet with p T > 25 GeV. This requirement is motivated by the fact that the target H ! 2⌧ had-vis + 2⌫ analysis is performed only in events with boosted ⌧ had-vis pairs [2]. Typically, a R o ine (⌧, ⌧) < 2.4 requirement is applied in the signal regions. Since geometric selections cannot be implemented in L1Calo, the L1 jet requirement is used to trigger on the jet activity recoiling from the boosted ⌧ had-vis pair. Such selection brings the rate down to 6.7 kHz and its e ciency in simulated Higgs boson events is shown in Figure 4(a) as a function of the R o ine (⌧, ⌧) between the two o ine reconstructed ⌧ had-vis candidates and the transverse momentum of the leading jet reconstructed o ine. The strong correlation between the e ciency as a function of R o ine (⌧, ⌧) and the e ciency as 10 Fig. 4. Comparison of the L1 2τ and 2τ +jet trigger rates as a function of the instantaneous luminosity for non-topological and topological triggers [5].
two taus, ∆R(τ 1 , τ 2 ) < 2.9, is introduced by the topological trigger rejecting a large fraction of minimum bias background events while retaining most of the signal events. As shown in Fig. 4, this reduces the trigger rate by a factor 4, a manageable level, without the need to raise the tau p T thresholds. The angular selection between the two taus has been proven to be crucial also when an additional jet with p T of 25 GeV is required. Also in this case the trigger rate is reduced by a factor 1.5 while maintaining the signal efficiency for the analysis. Topological triggers are useful also for other Higgs analyses. In the ZH → ννbb a requirement on the minimum angle between the E miss T and all the jets in the event allows to reduce the trigger threshold on the missing transverse momentum itself. Event hardness and invariant mass selections are used instead to trigger events in which the Higgs is produced via vector boson fusion. In particular, they try to identify the two jets from the scattering process instead of the Higgs decay products, in order to enhance the signal acceptance for the various analyses independent of the Higgs decay. This allows searches for Higgs with non standard model decays to also use these triggers. The trigger is designed to select a first jet in the central part of the detector (|η| < 3.2) and a second  jet in the full detector acceptance range (|η| < 4.9) forming a mass greater than 500 GeV. Additional requirements on the pseudorapidity and azimuthal angular separation between the two jets, ∆η jj > 4 and ∆φ jj < 2, are added at HLT level to further reject the QCD multijet background. The efficiency curve as a function of the offline maximum dijet mass is shown in Fig 5. The ATLAS B-physics program also strongly benefits from the use of L1Topo triggers. Many of these analyses are based on the identification of the B and J/ψ mesons via their decay products including very low p T electrons and muons pairs. For example, for the J/ψ → µµ analysis, the dilepton trigger for two muon trigger objects with p T of 6 GeV presents extremely high event rates. As shown in Fig. 6, by requiring the radial distance between the two muon trigger objects to be in the range 0.2 < ∆R(µ 1 , µ 2 ) < 1.5 and by requiring the invariant mass of the di-muon pair (m(µ 1 , µ 2 )) to be in the range 2-9 GeV, the trigger rate is reduced by a factor 3.5 with just a small loss in the signal efficiency. And, even more importantly, this does not introduce any bias or distortion in the mass distribution, which would prevent the use of such a trigger in the analysis. This allows a sustainable trigger rate while maintaining the p T thresholds of the muon trigger objects at their previous low values. The new selections and flexibility introduced by L1Topo also allows exploration of new ideas and possibilities. For example, it is known that the fixed ∆η × ∆φ sliding window method with size 0.8×0.8 used classically by the L1 jet triggers fails to capture all the energy of large jets with R parameter of 1 or larger. The method worsens with increased number of subjets in a jet, where it is more likely that a significant energy falls outside the selected range. The open points in Fig. 7 show how an increase of the number of subjets worsens the trigger efficiency turn-on as the full offline efficiency is reached at higher and higher jet p T . The L1Topo cone algorithm is designed to sum the transverse energy of all 0.8×0.8 windows with E T > 15 GeV within a cone of ∆R = 1. The obtained energy sum is required to be larger than 111 GeV to equal the rate of the non-topological 100 GeV jet trigger. The filled points in Fig. 7 show that the L1Topo cone algorithm is able to mitigate the effect of the energy loss strongly reducing the gap for jets with more than one subjet and recovering the trigger efficiency.

VI. CONCLUSION
The L1Topological processor allows the possibility to apply angular and kinematic selections already at the first level of the ATLAS trigger resulting in substantially improved background rejection, manageable trigger rates, and increased acceptance for signal events. It also adds new capabilities, like combining muon and calorimeter information, and objects from different bunch crossings. The system was successfully commissioned and routinely used for the full ATLAS Run-2. The development of simulation and validation tools were crucial to fully understand the hardware decisions and activate the topological triggers for physics analyses. The topological triggers were particularly crucial for physics analyses making use of low p T objects like in the Higgs and B-physics ATLAS program.