First results with the Gigabit Link Interface Board (GLIB)

We have designed and built an FPGA-based platform for users of high speed optical links in high energy physics experiments. The Gigabit Link Interface Board (GLIB) serves both as a platform for the evaluation of optical links in the laboratory as well as a triggering and/or data acquisition system in beam or irradiation tests of detector modules. The GLIB is a double width Advanced Mezzanine Card (AMC) that is used either stand-alone or inside a micro Telecommunications Computing Architecture (μTCA) crate. This paper presents the first results with the GLIB prototype delivered in 2011.


Introduction
We have designed and built a platform for the evaluation of optical links in the laboratory as well as for triggering and/or data acquisition systems in beam-or irradiation tests of detector modules. The Gigabit Link Interface Board (GLIB) [1] is based on an FPGA with Multi-Gigabit Transceivers (MGT) operating at rates of up to 6.5 Gb/s. This performance matches comfortably the specifications of the Gigabit Transceiver (GBT) [2] and Versatile Link projects [3] with its targeted data rate of 4.8 Gb/s. Figure 1 highlights the baseline configuration of a GBT -Versatile Link -GLIB system. On the left side, front-end (FE) ASICs are electrically connected to the GBT ASIC through e-links [4] while the GBT high-speed serial data-streams are converted to/from the optical domain through the Versatile Transceiver (VTRx) [5]. At the other end, the GLIB converts data to/from  the optical domain, implements the GBT protocol and codes/decodes the user payload at the link back-end.
This paper presents the GLIB prototype and reports on hardware tests, firmware development and system tests, software development as well as hardware development of auxiliary cards as part of the GLIB project.

Hardware
The GLIB is a compact board conceived in the form of a double wide Advanced Mezzanine Card (AMC) [6] that serves a small and simple system operating either stand-alone or residing inside a micro Telecommunications Computing Architecture (µTCA) [7] crate. Figure 2 shows a picture of the GLIB prototype highlighting its major components i.e. the Virtex-6 FPGA, the AMC connector, the cage for 4 Small Form Factor Pluggable Plus (SFP+) optical modules, the FPGA Mezzanine Card (FMC) [8] sockets, the SRAM devices, the socket for the Module Management Controller (MMC) mezzanine card [9] and the 1000Base-T interface.

Prototyping Experience
It is important to mention that the board shown in figure 2 is the second version of the GLIB. There were various reasons why we have designed a second version, even though the first version was relatively successful. The most important one was to correct various errors in the schematic design (e.g. failure to program the flash memory used for the loading of the FPGA after powerup, problems accessing one of the two SRAM device as well as some other minor issues). Other reasons were the decision to use a Ball Grid Array (BGA) package for the 1000Base-T physical layer device instead of the originally selected Bump Chip Carrier (BCC) because of assembly issues as well as the modification of the MMC socket in order to accommodate the latest version of the MMC mezzanine.

Results and achievements
This section reports on hardware and system test results obtained so far, the progress the firmware and software as well as on the development of auxiliary cards for the GLIB project.

High speed optical connectivity tests
For qualifying the high speed optical connectivity, we have measured the Bit Error Rate (BER) as a function of the Optical Modulation Amplitude (OMA) seen on the receiver side of the SFP+ module. The OMA is controlled by attenuating the optical output signal delivered by the transmitter side of the SFP+ while the BER is measured by using the built-in BER tester functionality of the FPGA. A Pseudo-Random Bit Sequence with 2 31 − 1 bits length (PRBS-31) has been used as the transmitted pattern for the tests. For all measurements, the same SFP+ module has been used. A diagram and a picture of the setup are shown in figure 3 (top left and bottom left, accordingly). Figure 3 (right) shows BER results obtained at 5 Gb/s 1 for all 8 optical channels of the GLIB in comparison to the measurements taken with the same SFP+ module housed on a different test board. The very good agreement between the BER measurements indicates clearly that there is essentially no deterioration of the optical module's performance due to the layout of the printed circuit board. It is also important to mention that there is a very large margin of error-free operation for these links since errors only appear after a very significant attenuation (approx. 16 dB) of the optical signal (the OMA of the SFP+ transmitter was approx. −3 dBm). The minor variations between locations are both due to small differences in the routing of the printed circuit and measurement repeatability errors.

High speed electrical connectivity tests
For the qualification of the AMC high speed serial connectivity, a commercial AMC breakout board carrying Sub-Miniature version "A" radio-frequency connectors (SMA) has been used. By connecting the GLIB to the AMC breakout board and using SMA cables to create loopback connections (figure 4, top left) we produce a fairly realistic test setup that is comparable with the connectivity path in a µTCA crate since the high-speed transmitted signals are travelling un-buffered through 2 AMC connections in both cases (real case and test setup 1 on figure 4, middle). Using this setup we have measured that the links can run error-free during tens of hours at 5 Gb/s (figure 4, top right). The performance deteriorated (error-free operation at up to 3.125 Gb/s) only after using an unrealistic setup ( figure 4, bottom) where the high-speed transmitted signals are travelling un-buffered through 4 AMC and 2 SMA connections (test setup 2 in figure 4, middle) by introducing an AMC extender between the two boards ( figure 4, bottom). It is worth mentioning that for defining the optimum configuration of important parameters for electrical high-speed links such as transmitter pre-emphasis and receiver equalization we have performed a parameter sweep procedure prior to the tests.

System tests
Concerning the FPGA firmware (figure 5), the development is currently focused on the stand-alone operation of the board controlled by a Gigabit Ethernet link. The GLIB firmware instantiates a simple IP-based control protocol (IPbus) designed for controlling xTCA-based hardware over Gigabit Ethernet that includes all basic transactions needed for this purpose (bitwise, single register and block transactions) [10]. The firmware also includes all interfaces to the external hardware e.g. I 2 C communication with the on-board temperature sensors and the serial memory, SPI communication with the clock synthesizer, SRAM interface (operating at up to 160 MHz) etc. etc. Additionally, the firmware instantiates a number of GBT Hardware Description Language (HDL) modules [11] that together with the associated Gigabit Transceiver (GTX) modules can be used for interfacing with GBT protocol based systems.
One of the important achievements of the project is the error-free communication between a GLIB that is clocked by an external reference with another GLIB that recovers the reference clock based on the received data-stream, thus simulating a back-end/front-end link where the "back-end" board is clocked externally while the "front-end" board is clocked from the incoming optical data (see figure 6). To achieve that, the procedure is the following: 1. The "back-end" GLIB transmits GBT packets.
2. The "front-end" GLIB starts-up using its local oscillator with a similar frequency to the external reference, recovers the clock from the data received, it feeds the recovered clock to the clock synthesizer and switches its input clock source. Now the board has exactly the same clock as the "back-end" GLIB.
3. The "front-end" GLIB transmits packets and the "back-end" GLIB receives them.
4. The link is up with two boards operating with exactly the same clock frequency.

Software development
On the software side, apart from the standard IPbus software framework distribution, we are developing a stand-alone JAVA-based Graphical Users Interface (GUI). This light-weight GUI provides to the users the ability to configure the board according to their needs in a simple way under any operating system. Screenshots of the GUI are shown in figure 7.     based on a commercial Clock & Data Recovery (CDR) integrated circuit for interfacing the GLIB with the TTC system used currently in the LHC experiments. This board is able to receive the 160Mbps bi-phase mark encoded bit-stream from an optical source, extract the clock and the data and forward them to the FPGA of the carrier card for further decoding of the data stream. Another board under development is the VTRx FMC, a mezzanine card that interfaces the various flavors of VTRx with the GLIB system. A third system which is currently being designed is an AMC to PCI Express x4 cable adapter, that provides a high bandwidth interconnection between a stand-alone GLIB and a PC.

Summary and status
The Gigabit Link Interface Board (GLIB) is an FPGA-based platform for users of high speed optical links in high energy physics experiments that is under development since September 2010. The first 2 GLIB prototypes have been built in early 2011. Already from the first prototypes, most of the board's features were functioning without problems. The few issues that have been identified, they have been understood and solved in the second version of the board that has been assembled in summer 2011. This version has been tested extensively and its bench-top performance has been found satisfactory. Currently, a preproduction of another 4 pieces is ongoing. We are planning to deliver a few of these boards to some selected beta users before the end of the year. Together with the hardware, FPGA firmware and associated software will also be delivered, including user logic examples. For the coming months we are planning to focus on the in-crate operation of the GLIB before going into production. The production version of the board is expected to be available in July 2012. Within the frame of this project, some additional auxiliary boards (mainly FMCs) are currently under development. The first prototype of the TTC FMC is already available while the Versatile Link FMC will be available by the end of 2011.
For the latest information concerning the GLIB, please visit the project's web site [12].