The Gigabit Link Interface Board (GLIB) ecosystem

The Gigabit Link Interface Board (GLIB) project is an FPGA-based platform for users of high-speed optical links in high energy physics experiments. The major hardware component of the platform is the GLIB Advanced Mezzanine Card (AMC). Additionally to the AMC, auxiliary components are developed that enhance GLIB platform's I/O bandwidth and compatibility with legacy and future triggering and/or data acquisition interfaces. This article focuses on the development of the auxiliary components that together with the GLIB AMC offer a complete solution for beam/irradiation tests of detector modules and evaluation of optical links.


Introduction
The Gigabit Link Interface Board (GLIB) [1] is an FPGA-based system for users of high-speed optical links in high energy physics experiments.The GLIB serves both as a platform for the evaluation of optical links in the laboratory as well as a triggering and/or data acquisition system in beam or irradiation tests of detector modules.The major hardware component of the platform is the GLIB Advanced Mezzanine Card (AMC) [2] that can be used either on a bench or in a µTCA [3] crate.The GLIB AMC is based on a Xilinx Virtex-6 FPGA with Multi-Gigabit Transceivers (MGT) operating at rates of up to 6.5 Gb/s.This performance matches comfortably the specifications of the Gigabit Transceiver (GBT) [4] and Versatile Link [5,6] projects with targeted data rate of 4.8 Gb/s.
Figure 1 illustrates GLIB-based system configurations.The baseline configuration of a GBT-Versatile Link-GLIB system is shown at the top.Front-end (FE) ASICs are electrically connected to the GBT ASIC through e-links [7] while the GBT high-speed serial data-streams are converted to/from the optical domain through the Versatile Transceiver [8].At the other end, the GLIB system converts data to/from the optical domain, implements the GBT data transmission protocol [9] and codes/decodes the user payload at the link back-end.An alternative configuration, useful for intermediate prototyping, is shown at the bottom with one GLIB interfacing to FE ASICs and VTRx, thus emulating the GBT, and a second GLIB at the back-end.
Figure 2 shows a picture of the production version of the GLIB AMC, highlighting the two high-pin count (HPC) FMC Mezzanine Card (FMC) [10] sockets.The presence of the HPC FMC sockets is a big advantage since they provide additional user-specific I/O, high-speed transceivers and clock lines that can be used to extend the I/O connectivity of the GLIB AMC.For that reason, most of the auxiliary boards developed for the GLIB platform adopt the FMC format.The purpose -1 - of the auxiliary boards is to enhance the GLIB AMC compatibility with legacy and future triggering and/or data acquisition interfaces as well as its I/O bandwidth when in bench-top operation.In addition to the auxiliary cards, a Graphical Users Interface (GUI) has also been developed in order to ease the platform's access and control.All hardware and software components developed around the GLIB AMC card are often referred-to as the GLIB "ecosystem" and will be described in the following sections.

GLIB ecosystem hardware 2.1 TTC FMC
The first auxiliary add-in board developed is a low-pin count FMC based on a commercial Clock & Data Recovery (CDR) integrated circuit for interfacing the GLIB AMC with the Trigger, Timing and Control (TTC) system used currently in the LHC experiments.This FMC mainly serves for bench-top systems, since when the GLIB AMC inserted in a µTCA crate, it can TTC information through the backplane.However, the TTC FMC could also serve in µTCA systems that do not incorporate special hardware (such as the CMS-designed AMC13 [11]) able to receive and forward TTC information over the backplane.
Figure 3 shows a diagram of the TTC FMC illustrating its main components.The board functions as following: it receives the 160 Mbps bi-phase mark encoded bit-stream from an optical source in one of its two optical inputs (ST type receiver or SFP socket), extracts the 160 MHz clock and data and forward them to the FPGA of the carrier card for further decoding of the data stream.Additionally, a dedicated chip divides the recovered 160 MHz clock by 4 providing an additional 40 MHz clock required for the majority of LHC-based systems.The determinism in the phase of the 40 MHz clock is ensured by the carrier's FPGA TTC decoding logic that controls the operation -2 - of the clock divider.The FMC includes as well an optional logic circuit that detects the status of the CDR and forces the cross-point switch to use the local oscillator as source when the CDR is not yet initialized.Moreover, the cross-point switch serves for selecting which two FMC clock lines to drive, making the TTC FMC also compatible with legacy FMC carriers. 1Figure 4 shows a picture of the latest version of the TTC FMC that is expected to enter production in early 2013.
Jitter analysis and decomposition were made on the TTC FMC mezzanine using an Agilent 91204A oscilloscope.The TTC signal was delivered to the TTC FMC through TTCvi and TTCex modules, and the clock source was delivered by an 81133A Agilent generator.Several channel occupancy scenarios were tested (idle TTC frame, 100 kHz trigger rate on channel A, or full channels A and B).The performance was very satisfying, with a TIE jitter of about 11 ps rms for an idle frame, and about 13 ps rms for a TTC frame with channels A and B continuously transmitting.The jitter measurements are summarized in table 1.

Versatile Link FMC
Within the context of the Versatile Link Project, radiation-hard optical modules such as the Versatile Link transceiver (VTRx) and the Versatile Link twin transmitter (VTTx) have been developed in order to convert GBT high-speed serial data-streams to/from the optical domain.In order to interface these modules with the GLIB AMC, we have developed the Versatile Link FMC.The first prototype of the board carries only the VTRx while the second version, which is currently under development, is able to host both the VTRx and VTTx modules.This functionality is very  Figure 5 shows pictures of Versatile Link FMCs.The Versatile link FMC is expected to be available in the 2 nd half of 2013.

E-link FMC
As mentioned in the introduction, future FE ASICs will be connected to the GBT ASIC through electrical serial links, known as e-links.Each e-link comprises three differential pairs (following the JEDEC SLVS signalling standard): one pair provides the clock driven by the GBT while the other two carry data information flowing in both directions.The e-links are specified to operate at -4 -

AMC adapter
In order to enhance the I/O bandwidth of the GLIB when used as bench-top board, we have developed an adapter card that carries an AMC socket, taking advantage of the high-speed serial links available in the AMC connector of the GLIB.More specifically, the AMC adapter card implements a quad-lane PCIe 2.0 over cable interface based on a PCIe repeater providing a high bandwidth interconnection with a PC equipped with a commercial PCIe host adapter card.The card also includes an additional GbE interface and an external power connector.Figure 7 shows the AMC adapter card attached to a GLIB.The AMC adapter is expected to be available in the 2 nd half of 2013.

GLIB ecosystem software
At the lowest level, communication with the GLIB can be established via two channels: PCIe and Ethernet.Access to these interfaces requires specialized S/W which has to be developed by skilled engineers.In order to provide users of the GLIB with a tool that enables them to exploit the basic functions of the board without going through such a development cycle and in order to provide code examples for users that want to implement their own S/W we have developed a Java based GUI for the GLIB which supports communication via both of the interfaces.This Java GUI consists of three main building blocks: the low level H/W access, classes for the handling of XML files and the graphical back end.

The XML classes
Each GLIB card comes with a generic IP core for its FPGA implementing the basic functionality of the board.This is the foundation on which users build their custom code.Both the core and the user code are controlled via 32 bit registers.The definition and location (in H/W address space) of these registers is not static.We have chosen to use an XML file for the modelling of the resources of a GLIB.The GUI package therefore contains classes that are able to un-marshal that XML file and to convert it into objects, representing registers or bit fields, which can be used by the H/W access code.A second XML file is used to record manual sequences of actions (reading and writing registers as well as delays) and to re-play them later on.

The low level S/W
This part of the S/W interfaces to the GLIB hardware and supports both Ethernet (UDP) and PCIe.
As the access to the resources of the GLIB is based on the IPbus protocol [12] there are classes that translate basic read and write operations into IPbus messages with proper framing.These classes are independent enough to be used for other (Java based) S/W projects but only provides a simple IPbus interface without support for command grouping.For access via Ethernet the whole code is in Java.As Java does not provide classes for the access to PCIe resources a device driver (io rcc) and an associated library (libio rcc.so), originally developed for the ATLAS TDAQ project, have been attached to the Java S/W by means of appropriate JNA wrappers.As the io rcc S/W is only supported for Linux, PCIe based access via the GLIB GUI is only possible if the GUI is run on a Linux based embedded processor AMC or desktop PC with a PCIe cable interface.

The graphical back-end
Once started on a suitable PC, the GUI will present the user with several panels (a snapshot of the GUI is shown in figure 8).Each of them provides functions of a certain category (simple register access, Status displays, list recording and replay, debugging tools).Users are able to add their own -7 - panels to the GUI framework.Detailed documentation of the GUI is available at [13].The entire S/W is available on a CERN SVN server and supported by the members of the GLIB team.

Summary
We reviewed the status of the GLIB ecosystem, a group of hardware and software components developed around the GLIB AMC card, which offers a complete solution for beam or irradiation tests of detector modules as well as for the evaluation of optical links in the laboratory.The production grade GLIB AMCs are just becoming available.FMCs follow, starting with the TTC FMC due to enter production in early 2013.Versatile Link and e-link FMCs as well as the AMC adapter are in prototyping stage and should become available in 2 nd half 2013.More details and up-to-date information about the GLIB ecosystem can be found in the GLIB project web page [13].

Figure 1 .
Figure 1.GLIB configurations.Top: a GBT-Versatile Link system with the GLIB at the back-end.Bottom: a GLIB interfacing to FE ASICs and VTRx with a second GLIB at the back-end.

Figure 2 .
Figure 2. Picture of the GLIB AMC, highlighting the two FMC sockets.

Table 1 .
Jitter measured on the 40 MHz clock recovered by the TTC FMC, for idle and full TTC frames.TTC FMCTTC FMC (TTC channels A&B idle) (TTC channels A&B full) allows the evaluation of Versatile Link components in the laboratory and more generally of GBT-based systems by integrating them in an FPGA-based environment as the GLIB.

Figure 4 .
Figure 4. Picture of the TTC FMC.

Figure 5 .
Figure 5. Pictures of Versatile Link FMCs: left: the first version carrying a VTRx.Right: the second version that can carry both VTRx and VTTx modules.

3
possible data rates: 80, 160 & 320 Mb/s.In order to interface e-link-based ASICs with the GLIB AMC we have developed the e-link FMC.This board has five micro-HDMI connectors in its frontpanel, each one providing two input and two output differential pairs as well as I2C interface and ground connections.The individually shielded differential pairs of the HDMI cables are specified for data rates up to 3.7 Gb/s, covering largely the maximum e-link data rate.In order to enable the -5 -

Figure 7 .
Figure 7.The AMC adapter card attached to a GLIB.

Figure 8 .
Figure 8. Snapshot of the GLIB GUI.