A prototype for the upgraded readout electronics of TileCal

Upgrade plans for the ATLAS hadronic tile calorimeter (TileCal) at the LHC include full granularity readout to the 1st level trigger. R&D activities at different laboratories target different parts of the upgraded system. We are developing a possible implementation of the future readout electronics to be included in a full functional demonstrator. This must be capable of adapting to each of the three different front-end alternatives being considered. Prototypes of the two PCBs that will be in charge of digitization, control and communication have been developed. The design is redundant and uses FPGAs with fault tolerant firmware for control and protocol conversion. Communication and clock synchronization between on and offdetector electronics is implemented via high speed optical links using the GBT protocol.


Introduction
This paper describes the upgrade plans for the ATLAS Tile Calorimeter (TileCal) readout electronics. The current TileCal readout electronics is responsible for digitizing PMT pulses and to provide a triggered readout. It is organized in units termed 'drawers' containing 96 front-end cards, called '3-in-1 cards', for shaping PMT pulses, 8 digitizer boards sampling the shaped pulses and a mezzanine card for receiving optical control signals to the front-end cards. These parts are carried on 4 motherboards per drawer. There is also an interface card for off-detector read out. The current readout electronics is described in detail in [1].
One drawer with the new readout electronics, dubbed the demonstrator, is scheduled to be inserted into TileCal during the 2013 shutdown. The new readout electronics will merge the functionality of the current electronics into a two board solution (the front-end cards potentially excepted depending on design path), Mainboard and Daughterboard. Analog signals are digitized on the Mainboard and transferred to the Daughterboard which houses control logic, translation logic and transceivers. The future readout electronics and thus also the full demonstrator will have 12 front-end channels per board and 4 boards per drawer. All data will be read out continuously which requires a data rate of 19.2 Gbit/s per board or 38.4 Gbit/s with 2 fold redundancy. The transceiver solution of choice is the SNAP12, a pluggable 12 parallel fiber bundle. The SNAP12 was chosen as a compact way of testing many links. The final transceiver solution is yet to be decided. The demonstrator will still have analog outputs to an analog trigger system in order to be compatible with the current system. There are at the moment three alternatives for the front-end electronics and the first version of the Mainboard is adapted to one of them, the "3-in-1" alternative. The daughterboard should be the same, regardless of front-end solution. Later versions will be adapted to the other two.
The prototype described here is limited to four channels. It has also been designed with some parameters to be optimized experimentally in actual operations. These include terminator and decoupling values.

Mainboard
The Mainboard prototype is responsible for digitization of the front-end signals and also acts as a carrier for the "3-in-1" front-end cards. It has 4 channels, organised in two symmetric sections. Off board communication is done via an FMC HPC connector using the VITA 57 pinout.
The pulse digitization is performed by four LTC2264 dual 12-bit ADCs. Each front-end signal is sampled in high gain and low gain, both signals provided by the '3-in-1' cards, to increase resolution without requiring extreme noise performance. One DAC for each section provides programmable pedestals for the inputs.

ADC
The LTC2264 is a 12 bit dual ADC with serial outputs. The sampling clocks, which operate at 40MHz, are produced by the Daughterboard. The serial outputs are differential and running at 480Mbit/s. The voltage range is programmable and for this application set to 2V, giving us an LSB of 0.5mV. Special attention has been given to the input filtering and grounding, both for performance and evaluation purposes. The ADC inputs have a common pedestal level for each section, controlled by a DAC. This is programmable via the daughterboard.

Power
In order to test noise performance, the Mainboard does not rely on local regulators, except for the 1.8V needed by the ADCs, and has ample space for filters. We have also designed one section of the board with separated analog and digital ground planes and one section with common ground planes. The Mainboard also delivers power, 3.3V, to the Daughterboard through the FMC connector.

Daughterboard
The Daughterboard is responsible for distribution of commands and clocks to the Mainboard and readout to off-detector electronics. It's also intended to serve future Mainboard prototypes. For redundancy reasons there are 2 independent sections essentially making it two boards in one with independent functionality and power solutions. Each section has logic based on the Virtex 6 FPGA (XC6VLX130T, the second smallest chip in the Virtex 6 series), with serial configuration memories (SPI). There is a crystal oscillator as well as a dedicated differential SMA clock input, though the design is intended to use the recovered TTC clock for all operations. The firmware has already been validated in previous work [2].
The board requires 4 voltages, 3.3V, 2.5V, 1.2V and 1.0V. The FMC connector provides 3.3V which also feeds onboard regulators used for the other voltages. The supplies for the optical transceivers have Π-section low pass filters near the components. The main power draw on the board are the FPGAs. With 6 GBTs implemented they can draw up to 5W per FPGA, which will put power consumption of the board above the VITA 57 FMC standard (10W without cooling), but within the limits of connector capacity. The full Demonstrator will require more power pins in the FMC connector than the standard.

Connectivity
The Daughterboard offers 59 differential I/O lines and 17 single ended I/O lines through the FMC connector. All routed to both FPGAs using a multipoint topology to reduce connections between the boards. Optical links (2xSFPs and one Snap-12) are used to transmit clocks, data, commands and signals encoded with the GBT(GigaBit Transceiver)-FPGA protocol [3] at 4.8 Gbit/s/link.
The Daughterboard prototype is adapted to the Mainboard prototype and must therefore fit the Mainboard connectors. However, for test purposes it was considered important to make the first prototype compatible with other general FPGA boards, using the VITA57 FMC pinout. This was achieved by temporarily accepting some limitations. Thus, several leads in the FMC are not yet -3 -connected and some lines, defined as differential in the specification, are routed single-ended. This will be revised in later versions when the functionality is fully proven. The redundant connections to the FPGA are a major design constraint and impact the size of the FPGA. Foregoing redundancy, i.e. having half the board map to half the Mainboard, would significantly lessen this constraint.

Redundancy
As mentioned above, the board is divided into two independent sections for redundancy. Each section has its own power but the power distribution can also be connected in parallel. The only single point failure node is the Avago SNAP12. While the 12 channels are independent they share a control interface. This is a single point failure node not only through damage to the transceiver but also since we need a master controller. Statistically the risk is small but a failure will mean the loss of a quarter of a drawer.
For the firmware we are using triple mode redundancy, TMR, where cost effective. The extent and depth of TMR is a design specific choice as well as a matter of available software. The Virtex 6 also offers built-in error correcting block RAM, ECC BRAM. The effectiveness of this functionality will be studied in future radiation tests.

Tests
The Mainboard functionality has been verified with a XILINX ML605 FPGA evaluation board. The communication between Daughterboard and Mainboard has been verified and we have successfully configured ADCs from the Daughterboard.
The multipoint topology is also working which means we do not need external logic to split signals to both FPGAs. A few single ended signals were routed with large stubs and suffer a penalty for it but this is not an issue for the intended slow application of these signals. As previously mentioned, the single ended lines will be addressed in later versions.
The Mainboard is performing as expected, with room left for further improvements of the filters. The following measurements were made with Chipscope and are measured on the side with common analog and digital ground. Figures 4 and 5 show a composite of 4 pulses, each shifted 90 • , from a "3-in-1" front-end card, sampled at 40MHz. The ground bounce after the pulse is not severe and can be reduced further by optimizing the input filters. Figure 6 and 7 show the LSB noise resolution measured on a bias voltage. The measurements were taken with front-end boards connected, which is why there is a difference between low and high gain. Noise performance is good despite not having optimized filters on the power supplies.
Eventually the entire demonstrator must be exposed to relevant radiations to verify that it can survive in its intended environment and recover from single event upsets. Several components have already been tested. The Chicago group has validated the '3-in-1' cards and are working on radiation tests for the Avago AFBR-775 [4]. The power regulators will be exchanged for ones from the ongoing effort at CERN to certify regulators for radiation environments. Testing the FPGA firmware in radiation remains to be done.    The '3-in-1' version Mainboard is ready to be scaled up to a full demonstrator. Other front-end options may require additional work. As with the Daughterboard, local power regulators will be chosen from the CERN effort. We also plan to use PPTC, polymeric positive temperature coefficient, fuses in the next version. These are self resetting fuses, ideal for conditions with limited or no access.

Daughterboard
A crucial feature of the demonstrator is the in-system programmability (ISP) of the FPGAs. This will give us the possibility to test and upgrade designs as well as a way to recover from errors in the configuration memories due to radiation. The intention is to use the rad-hard GBT chipset being developed at CERN for this purpose but since the chip was not available at the time of production -5 -we used SFPs directly to the FPGA for downlink communication. Since we have already demonstrated JTAG via CPLD before and the implementation is straight forward, adding JTAG via GBTX to the next version should be straightforward.
The next version will have all I/O lines on the FMC connected, requiring denser routing and/or more layers. The routing on the prototype is conservative, simulation shows a worst case crosstalk around 10mV on the 480 Mbps lines, so it is certainly possible to do a denser solution. Using materials with higher dielectric constant would also allow denser routing.
The board can also be made more compact. Components are currently spaced further apart than needed, which was a deliberate choice to make testing and debugging easier. Radiation testing especially favours well spaced components.

Summary
We have developed a prototype readout test system for TileCal. The boards are communicating and we can read out pulses from the front-end electronics. Compliance with the current TTC system is yet to be tested. The overall performance is satisfying and the few bugs left to work out are minor. With the information gathered we are ready to proceed to a full demonstrator design of the Daughterboard and a '3-in-1' Mainboard.