The ATLAS Tile Calorimeter digitizer

The ATLAS Tile Calorimeter digitizer system samples photomultiplier signals from the scintillating tiles of the hadronic calorimeter. For each channel a pair of 10-bit ADCs digitize high and low gain signals at 40.08 MHz to provide the necessary 16-bit dynamic range. The sampled data is temporarily stored in digital pipelines for up to 6.375 ms, awaiting a level-1 accept. For each accept received, the corresponding sampled pulse is transferred to a derandomizer buffer for subsequent readout to the data acquisition system (DAQ).


Introduction
The Atlas hadronic Tile Calorimeter [1], TileCal, consists of 4 barrel-shaped segments, each containing 64 wedge-shaped modules (figure 1). The modules consist of interleaved iron plates and scintillating plastic tiles. Interacting particles entering the calorimeter produce showers of charged particles in the iron, which pass through the scintillator tiles, producing flashes of light. Wavelength shifting fibers carry the light to photomultipliers (PMTs) at the base of each module, installed in removable electronics "drawers". Each PMT collects light from a limited volume, a calorimeter cell, but each cell sends light to two PMTs.
The PMT signals are shaped and amplified in so-called 3-in-1 boards [2] next to each photomultiplier. Two output signals, high-and low-gain with a gain ratio of 64, provide the input for the digitizer system. The factor 64 allows two 10 bit ADCs to cover the 16 bit dynamic energy range (15 counts representing 0.5 GeV from a muon to 2 TeV from a high energy jet) without impacting the calorimeter resolution [3]. The 3-in-1 boards are also designed to provide well-defined charge injection pulses for calibration purposes.
The Tile Calorimeter digitizer system, also located in the electronics drawers, digitizes all PMT-signals continuously and stores the data in a temporary buffer. Only selected data sequences (time frames) are transferred to the data acquisition system via the readout interface [4] and the Tile Calorimeter ROD system [5], which is the first station of the second level trigger. The selected time frames are first transferred to a derandomizer buffer for readout on a first come, first served basis. During readout they are formatted according to a specified protocol, including headers for identification and trailers for error detection.
The digitization and processing are synchronized and controlled by signals from the TTC system [6] distributed via optical fibers. From this a continuous 40.08 MHz clock signal and a level 1 accept (L1A) signal is extracted. The L1A is the signal that triggers the selection of a time frame. Apart from the clock and the L1A the TTC signal also transmits broadcast and addressed commands to the digitizers for configuration and control.

JINST 3 P01004
-2 -  The production version of the digitizer [7], completed in 2001, was the end result of a series of prototype designs [8], [9], [10]. These were developed and tested in several TileCal test beam runs, where they also served as the readout channel for the test data. The design and its specifications evolved considerably during this process. The original ideas were derived from the LHC R&D project RD-16 [11] (FERMI, or Front-End Readout Microsystem), aimed at the development of a fully digital readout system.
The system serves 45 channels (calorimeter cells) per module in the two inner "barrel" segments of the calorimeter, and 32 per module in the outer, "extended barrel". Each 318 × 100mm 2 digitizer circuit board serves up to 6 channels. Inside a typical barrel drawer a row of 8 digitizer boards provides the necessary functionality, while 6 boards are sufficient for an extended barrel drawer.

Functional specification
The incoming pulses, high gain and low gain, are digitized every 25ns by 10-bit ADCs, using a sample clock that can be adjusted collectively for all ADC in units of 106 ps. Adjustment is necessary to ensure that one sample is always taken at the maximum point. The data is stored temporarily in pipeline memories awaiting an L1A. The pipeline latency is programmable up to 6.375 µs (2.5µs stipulated). At each L1A a time frame containing up to 16 samples is transferred to a derandomizing buffer for readout. The programmable time frame length up to 16 sample points is more than adequate considering the ~100 ns shaped pulse width. All pulses corresponding to the triggered event are later digitally processed in the read out driver (ROD) system, the correct amplitudes extracted and the correct time identified, eliminating, as far as possible, contributions from later pulses (pile-up). The derandomizer buffer in the production version of the digitizer can fit 16 16-word time frames (the memory size is 256 words). If smaller time frames are used the number of time frames will increase accordingly. Headers and CRC trailers are added to the data for readout. In the standard readout mode, only one of the high-and low-gain time frames is read out, to reduce bandwidth. The high gain channel is read out by default unless it overflows or underflows.
An important consideration is that the digitizers are located in an area with moderate radiation levels, increasing towards the gap between barrel and extended barrel. This makes it necessary to use only qualified radiation tolerant components or commercial off the shelf components (COTS), verified by appropriate radiation tests.
A crucial issue when designing a digitizer system is to reduce digital and other noise contributions below one least significant bit (LSB). In our case this was simplified by the fact that although the dynamic range is 16 bits we use only 10-bit ADCs. This means that the relevant LSB is not as critical as if we had used one 16-bit ADC. A low noise level was achieved by using differential inputs, and careful design and layout of the circuit board. Analog and digital power and ground are well separated (figure 2) and connected only at one point. Other noise reducing actions is that the power transport through the circuit boards for providing power to downstream boards occurs on dedicated layers and that there are additional power filters in form of ferrite beads on all power inputs. The system noise is also reduced by the fact that we use LVDS for digital signal transmission between boards.

Digitizer system architecture
The 8 board chain is divided into two parts, 4 boards each, connected with power and signal connectors (figure 3). Power enters at the mid point of the module and flows through the boards to the extreme ends, where on one side sense wires are attached. This flow also applies to the trigger, timing and control (TTC) signals. The digital output data flows in the opposite direction, towards the centrally located interface board. The central location for the readout is favored due to the lower radiation levels compared with the ends of the drawer.
Output data is sent simultaneously from all boards in a semi-serial (see below) form, using only four lines for each board to reduce to 32 the number of output lines reaching the interface board. Although the boards are connected sequentially they are functionally arranged in a star

JINST 3 P01004
-4 -configuration, where signals that must pass neighboring boards on their way to the interface board are re-mapped without involving active components for each board they pass (see the bottom part of figure 3). In this way, a board failure will not affect adjacent boards. Surface mount connectors are used for the output signals, and impedance matched flex foils transport the differential LVDS signals between the boards. The circuit boards and the flex foils are robust conveyors of the signals. However, this is not the case with the connectors (Panasonic AXK5S00047YGJ and AXK6S00447YGJ). These are a sensitive part of the design. They must be connected and disconnected carefully and when connected, fixed rigidly. A connector failure may affect all the boards outside the connector. The same is true for the power connectors.
The production board contains 6 layers, 3 for signal routing and 3 for power. A specialformat small board with only 3 channels was produced for a truncated drawer. In the outer part of the extended barrel, 3 boards are required to reach the calorimeter cells but only 2 are needed to service the channels there. Thus, a dummy board was used for the first of the 3 boards.
The analog part of the board contains components providing a voltage reference for the Analog Devices AD9050 ADC. It also contains two 8-bit DACs that provide a well-defined pedestal for the AC-coupled inputs. The analog part is duplicated so that each part only serves 3 channels.
The digital part of the board contains the two TileDMU gate arrays, the TTC receiver and decoder chip (TTCrx) [12], some LVDS driver and receiver chips, and a resistor field for hardwiring the board address. The TTCrx extracts the clock, L1A and commands, maintains the Bunch Crossing Identifier counter (BCID) and manages the phase adjustment of two phase adjustable clocks, clk_des1 and clk_des2.

The Tile Data Management Unit
All non-standard digital functions are implemented in two identical circuits per board, the TileDMUs. These are implemented in a 0.35 µ CMOS gate array (CX3061 from Chip Express [13]) manufactured in a radiation tolerant process to ensure that the entire system is sufficiently radiation tolerant. The design was specified in VHDL, and routed and verified using relevant chip delay from the manufacturer. The function was then physically verified against a laserprogrammed prototype chip.
Data from six ADCs (three high gain and three low gain channels) enter the TileDMU in parallel (figure 4) using the ADC_clock (clk_des2 from the TTC) which can be phase adjusted within one clock cycle. To avoid set-up and hold problems, a register that can be clocked either by the ADC_clock or its inverse is included between the two clock regions.
Four parity bits are added before the data is entered into the pipeline memory. On a L1A, a time frame including some pre-samples and a number of samples covering the pulse is transferred to the buffer memory. The start address of each time frame in the buffer memory is stored in an address FIFO, along with the corresponding BCID. At the same time the time frame is checked to determine whether any of its members are outside a specified range, to produce a gain flag to be stored in a flag FIFO.
When a readout is begun, a header is built and sent to the serializer. The next buffer address is retrieved from the address FIFO along with the corresponding gain flag, which is used to select the gain when in normal mode. This procedure allows overlapping time frames. Depending on the operating mode, either one or both gains are sent to the serializer. When reading data from the derandomizing buffer parity bits are checked and stripped. Later parity is -5 -added to the output data words. If a memory parity error was encountered the corresponding parity bit in the output word will be reversed, forcing a parity error condition. At the end of the 2-bit wide data sequence two CRC-16 trailers are appended. The ambition has been to catch any bit error occurring in the data path of the digitizer system. Close to full coverage has been achieved.

Commands and data formats
The digitizer is controlled by broadcast or addressed commands from the TTC. Most commands are interpreted in the TileDMU, i.e. the ones that are not directed to the TTC-rx itself. Using the commands it is possible to adjust the pipeline length (1 to 255), the number of samples in the time frames (1 to 16) and fine tune the timing of the ADC clocks in units of 106 ps, as well as choose the setting of the two pedestal DACs. The unit 106 is chosen for implementation reasons. Another command determines whether to invert the ADC clock at the receiver register (see above), which output speed mode to choose (20,30,40 MHz or handshake) and whether to insert wait states after the header word. The flexibility in choosing output rate and wait states

JINST 3 P01004
-6 -was important during development, but in ATLAS only 40 MHz without wait state was used. Each digitizer supports a continuous readout rate of 20 MB/s of which about 10% is used in a normal run. An S-link protocol [14] is provided for communication with the interface board, with a link control signal to flag link start (11) or link stop (00) commands, with data valid, link test and link reset. One can set the operation mode to normal mode, calibration mode or test mode. In the normal mode the entire time frame is compared to a programmable 10-bit range with an upper and lower limit. If the data is outside this range low gain data is sent. Otherwise, high gain data is read out. This reduces the required bandwidth by a factor of two compared to the calibration mode where both high gain and low gain data are sent.
In test mode the pipeline memory is filled with a programmable 30-bit field, which rotates 1 bit from cell to cell. The TTC system can be configured to send user-activated trigger when the system is in test mode.
In both normal and calibration modes, data from the TileDMU is read out serially on two lines, event-by-event, with channels grouped three-by-three. All boards are read out simultaneously delivering 32 bits to the interface each clock cycle.
The header contains flags (figure 5) to identify the data, such as a bunch crossing identifier, readout mode and number of samples. It also contains information about the gain setting for each ADC value and about the operation mode. Error flags monitor the digitizer operation and the TTC transmission. Single and double bit TTC errors are recorded. One flag is the combined parity of all configuration registers and other flags give the instantaneous 4-bit residue of all internal pointers. These bits are intended as a tool to verify that similarly programmed TileDMUs remain synchronous, i.e. that none of the control or pointer registers have been altered by a spurious (radiation induced) fault.

Quality assurance
The production version of the TileCal digitizer was finalized after thorough prototyping and was subjected to ATLAS standard design review process where many suggestions helped to improve the performance. An important part of the development was the radiation testing. Since the maximum expected radiation exposure is 200 rad/year of ionizing radiation and a neutron fluence of 10 11 1MeV-eq-neutrons/cm 2 /year, the radiation tolerance requirements was set to 10 krad of ionizing radiation and a neutron fluence of 5 . 10 12 1MeV-eq-neutrons/cm 2 to cover 10 years of LHC operation plus a suitable safety margin. The components were tested for ionizing radiation in Stockholm, neutrons in Debrecen and charged particles in Uppsala [15]. The last test was done while reproducing normal operating conditions in a test bench. We never saw any permanent component failures but in the last test we saw transient errors, due to soft single event upsets (SEU) corresponding to a bit error rate (BER) of about 2 . 10 12 . These were detected by parity and CRC. At some occasions we saw a stepwise strongly enhanced BER, from which one could recover after a system reset. This was probably due to soft SEUs in control structures. The manufacturers routinely tested the boards and the flex foils for continuity and shorts. After mounting components in a commercial company the boards were tested, burned in for one week at 70 o C and then tested again at Stockholm University using different test benches [16]. Functional boards were sent to Clermont-Ferrand for assembly into drawers and further tests. Tests were made after installation in the modules [17], [18], after the modules were installed in TileCal [19] and then at many occasions subsequently.
Most of the digitizer faults uncovered by the repeated tests can be attributed to connector problems. Some power issues were solved by replacing the initial spring-contact based connectors with screw-tightened ones, and proper seating of the surface-mount flex foil connectors was ensured by screws and washers or by supporting collars [20] so that the connector was firmly fixed but not mechanically stressed. Other problems were related to the fragility of the surface mount connectors, and difficult conditions when replacing components in the detector.
At a lower rate of incidence there were also problems related to the digitizer boards themselves. Most common were faulty solder joints but component failures also occurred. A mechanical stress test or thermal cycling would have uncovered many of these. This would have been more useful than the burn in, which did not provoke many malfunctions.

Summary and conclusions
A 40 MS/s digitizer system for the ATLAS Tile Calorimeter, comprising of close to 10000 channels, has been designed, prototyped, tested and verified. A critical issue was radiation tolerance, which was achieved by using a combination of radiation tolerant components and standard components proven sufficiently radiation tolerant in tests.
After a recent drawer refurbishment campaign followed by a limited operation time in its production environment, but so far without radiation, the digitizer system appears reliable with no digitizer board or DMU single point failures. So far we can estimate that less than 2% of the digitizer modules and less than 1% of the DMUs will fail in a year at a 90% confidence level. This is based on 6.4k digitizer observation days corresponding to 117 k DMU days. However, the estimate improves continuously with increasing time experience [21].
It is most probable that digitizer system must be modified for operations in the event of an LHC upgrade (SLHC). One of the critical points will be to verify that it will tolerate the increased radiation levels. Larger luminosity will increase pile up unless the shaping time is reduced. Reduced shaping time will require higher sample rate (probably 80 MHz) and therefore, new ADCs. The design will probably also have to adapt to a new version of the TTCrx. In this case, the TileDMU will have to be replaced, preferably by a programmable component, if such a component can be found with sufficient radiation tolerance. However, the overall digitizer concept should still be viable.