CMS Bulletin

DAQ

The DAQ system (see Figure 2) consists of:
– the full detector read-out of a total of 633 FEDs (front-end drivers). The FRL (front-end readout link) provides the common interface between the sub-detector specific FEDs and the central DAQ;
– 8 DAQ slices with a 100 GB/s event building capacity – corresponding to a nominal 2 kB per FRL at a Level-1 trigger rate of 100 kHz;
– an event filter to run the HLT (High Level Trigger) composing 720 PCs with two quad-core 2.6 GHz CPUs;
– a 16-node storage manager system allowing a writing rate that exceeds 1 GB/s, with concurrent transfers to Tier 0 at the same rate, and a total storage capacity of 250 TB. It also forwards events to the online DQM (Data Quality Monitoring).

Figure 2: The CMS DAQ system. The two-stage event builder assembles event fragments from typically eight front-ends located underground (USC) into one super-fragment which is then FED into one of the eight independent readout slices on the surface (SCX) where the complete event is built.

Developments for the 2011 physics run

A number of releases and updates of the online software, including framework and services, run control and central DAQ applications, have been made. These addressed bug fixes, performance improvements and functionality enhancements.

The XDAQ framework has been consolidated for the SLC4/32-bit platform with release-10.  This release is targeted to the sub-detector DAQ nodes and all those nodes have been migrated to the latest update at the beginning of 2011. Furthermore, release-11 has been developed, which supports in addition to the SLC4/32-bit platform, the SLC5 platform (64-bit OS, 64-bit applications with the alternate gcc434 compiler). The SLC5 platform is targeted to the central DAQ nodes, and in particular the HLT, Storage Manager and online DQM nodes running CMSSW.

All central DAQ nodes have been migrated to SLC5/64-bit kernel and 64-bit applications. The HLT, Storage Manager and online DQM are running currently CMSSW 4.x and the executables are built with the same compiler as offline. There is a ~20% performance improvement for HLT from the move from 32 to 64 bits.  The sub-detector DAQ nodes – mainly used for control and configuration of the VME crates – have stayed on SLC4/32-bits for the time being.

The HLT farm has been extended with additional PCs to increase the HLT power by about 50%. The cooling of the online data centre in SCX5 has been upgraded from 600 kW to 1 MW during the technical stop at the end of 2010. This increased cooling capacity allows for the current and all future extensions of the DAQ-HLT system.

Figure 3: The EVB-HLT installation.

The DAQ system deploys a two-tier event builder (see figure 2): an initial pre-assembly by the FED builder on the first stage, and a final assembly and event selection by eight independent DAQ slices on the second stage. Each FED builder assembles data from (typically) eight FEDs into one super-fragment using Myrinet switches. The super-fragment is delivered to one of the eight independent DAQ slices on the surface, where it is buffered in readout units (RUs) running on commodity PCs. Each readout unit (RU) is connected via a 540-port switch to Builder Units (BUs) using the TCP/IP protocol over Gigabit Ethernet. The event building in each slice is controlled by one event manager (EVM), which receives the L1A trigger information. The BUs store the complete events until the filter units (FUs), running the HLT algorithms, either reject or accept the events.

Data of accepted events are compressed and sent to the storage manager, which writes them to disk. The switching-fabric of the second stage event builder composes eight Force-10 E1200 switches (one for each DAQ slice), with a grand total of 4320 1 Gbps ports.  The PC nodes used during 2009-2010 for the combined BU-FU function are 720 units of Dell PE 1950 dual quad-core, 2.66 GHz CPU (Intel E5430 “Harpertown”) and 16 GB memory. All the nodes have two data links to the switching fabric, as there needs to be sufficient data bandwidth to sustain an event rate per node of 139 Hz with a L1 trigger rate of 100 kHz. At the beginning of 2011 an extension of the HLT farm has been installed (see Figure 3) by adding 72 units of Dell PE C6100. These compact units house four system boards, which are independent computing nodes, in a 2U chassis with a shared power supply. A configuration with dual six-core, 2.66 GHz CPUs (Intel X5650 “Westmere-EP”) and 24 GB memory has been chosen. These CPU cores support hyper threading which might give an additional performance improvement. This has brought the total HLT system to 1008 nodes, 9216 cores and 18 TB of memory. The estimated available CPU time for HLT processing has been increased from ~50 ms/event to ~82 ms/event. It is expected that the HLT extension will be fully commissioned and integrated during the April technical stop.

Based on experience with the 2011 data, a decision will be made if more HLT processing power is required for the 2012 LHC operation. In that case, a further extension of the HLT is possible by installing additional HLT nodes and re-cabling the event building network to reduce the number of data links per node from 2 to 1 and distribute the ports to the larger number of HLT nodes.