Addressing Scalability with Message Queues: Architecture and Use Cases for DIRAC Interware
- Krzemien, Wojciech et al
- arXiv:1902.09645
 | In DIRAC, the MQ configuration is defined as resource sections set through a DIRAC Configuration Service (not in the figure). Each section is uniquely identified by the pseudo-url string, e.g., ``mardirac3.in2p3.fr::Queue::Q2'', which can be provided as an argument to the factory methods createConsumer() and createProducer() responsible for the creation of producer and consumer instances. To improve the performance, the same connections can be reused by several consumers or producer. This functionality is provided by MQConnectionManager module. The purpose of MQConnector interfaces is to provide a mechanism that accommodates various communication protocols e.g. STOMP~\cite{stomp}. More details are given in the text. |
 | Message-queueing asynchronous communication model. Producers send messages that are stored by the intermediate component (Message Queue). The messages can be retrieved by consumers. The Message Queue decouples interacting entities. |
 | Architecture for the universal pilot logging system. The MQ broker (in the scheme ActiveMQ~\cite{activemq}) collects the log information provided by the pilot. The transfer is performed via the Tornado server~\cite{tornado}, that assures both authentication and authorization. The MQ consumers receive the logs that are finally stored by a dedicated DIRAC service. |