Search for Rare Decays of the Observed Higgs Boson and Additional Higgs Bosons with the ATLAS Detector

Searches for rare decays of the Standard Model Higgs boson and additional Higgs bosons are performed by the ATLAS experiment at the LHC. They use the unprecedented amount of collision data collected during the LHC \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$pp$$\end{document} collision run at \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sqrt{s}=13$$\end{document} TeV in 2015–2018, corresponding to an integrated luminosity of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$139\text{ fb}^{-1}$$\end{document} after the data quality requirements. For the search for the dimuon decay of the Standard Model Higgs boson, the observed (expected) significance is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$2.0\sigma$$\end{document} (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$1.7\sigma$$\end{document}), and the best-fit value of the signal strength parameter is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu=1.2\pm 0.6$$\end{document}. For the search for the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Z\gamma$$\end{document} decay of the Standard Model Higgs boson, the observed (expected) significance is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$2.2\sigma$$\end{document} (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$1.2\sigma$$\end{document}), and the best-fit value of the signal strength parameter is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu=2.0^{+1.0}_{-0.9}$$\end{document}. For the search for new resonances decaying into photon pairs, no significant excess is observed. The experimental sensitivity of these analyses to the signal processes is increased, due not only to the increase in the dataset but also to the improvement in the analysis techniques.


INTRODUCTION
The Standard Model (SM) of elementary particle physics is established with the discovery of the Higgs boson in 2012 by the ATLAS and CMS experiments [1,2] at the LHC. However, there are reasons to believe in the existence of new physics beyond the Standard Model (BSM), for instance the hierarchy problem.
Searches for unobserved processes in the Higgs sector provide important probes for new physics phenomena. In this paper, results for the search of rare decays of the SM Higgs boson (H → μμ, H → Zγ) and of additional resonances decaying into photon pairs are presented. These results are based on the entire Run 2 dataset of the ATLAS experiment [3] collected in the years 2015-2018, that correspond to the integrated luminosity of 139 fb −1 after the data quality requirements.

SEARCH FOR THE DIMUON DECAY OF THE STANDARD MODEL HIGGS BOSON
The Yukawa coupling of the Higgs boson to the third-generation charged fermions (t, b, τ ) and weak bosons (W, Z) are already measured, but the decays into the second-generation charged fermions are unobserved. The search for the H → μμ process provides a unique opportunity to observe the coupling * E-mail: yuya.kano@cern.ch to a second-generation charged fermion at the LHC. Additionally, measuring the branching ratio B(H → μμ) serves as a probe to new physics, because the existence of new physics may alter the branching ratio [4,5]. The decay H → μμ is a rare process with a SM prediction for the branching ratio of B(H → μμ) = (2.17 ± 0.04) × 10 −4 .
The signal-to-background ratio is as low as S/B ∼ 0.2% in the analysis signal region near the dimuon mass peak 120 GeV < m μμ < 130 GeV, with the Drell-Yan process being the dominant background component. This makes the observation of such process challenging. The analysis [6] first selects events with two high-energetic reconstructed muons that are isolated and have opposite electrical charge. Adding up to one final-state-radiation photon to the calculation of m μμ improves the mass resolution by 3%. The events in the analysis signal region are categorized into 20 mutually exclusive categories. They are defined to target different production modes of the Higgs boson, namely the top-quark-pair associated production (ttH), vector-boson associated production (VH), the vector-boson fusion (VBF), and the gluon-gluon fusion (ggF ) processes. They are based on the number of reconstructed highenergetic jets, the number of reconstructed highenergetic leptons (e or μ), and discriminants of boosted decision trees (BDTs). The BDTs target the kinematic differences between the background processes and the different Higgs boson production modes.
The signal extraction is performed by simultaneous fit to the m μμ distributions of all event categories. The signal component is modelled by a double-sided Crystal Ball function, which has a Gaussian central part and power-law tails. The background component is modelled by a product of the core function and the empirical function. The core function is a leading-order (LO) Drell-Yan line-shape that is convolved with detector effects (assuming a Gaussian function). The empirical function is an analytic function with free parameters, that is used to describe the distortions in the shape of the m μμ spectrum due to, e.g., the kinematic selections, event categorizations, and contributions from minor background components. The validation of this backgroundmodelling approach is performed using templates of m μμ distribution that are obtained using simulated samples of the Drell-Yan process, produced using the fast-simulation procedure to obtain a high statistical precision. This is illustrated in Fig. 1.
The m μμ distribution in the analysis signal region and the fit to the spectrum is illustrated in Fig. 2. The observed (expected) significance with respect to a null hypothesis is 2.0σ (1.7σ). The best-fit value of the signal strength parameter μ obtained from the statistical analysis is 1.2 ± 0.6, where μ is defined as the ratio of the observed signal event yield and the SM expected one. The uncertainty of the signal strength parameter is dominated by the statistical error, which correspond to ±0.58. The obtained limit on the signal strength parameter is μ < 2.2 at the 95% confidence level.
Overall, the expected sensitivity improves by a factor 2.5 compared to the previous ATLAS publication using a dataset corresponding to 36 fb −1 [7]. Of this improvement, a factor of about two is due to the larger size of the dataset, and a factor of about 25% is due to the improvement in the analysis techniques.

SEARCH FOR THE Zγ DECAY OF THE STANDARD MODEL HIGGS BOSON
The decay H → Zγ is a rare process with a predicted SM branching ratio of B(H → Zγ) = (1.54 ± 0.09) × 10 −3 . It proceeds through loop diagrams. The measurement of the branching ratio is sensitive to new physics, for instance the contribution of new charged particles to the loop [8][9][10].
The search [11] is performed in the eeγ and μμγ final states. This targets the decay of the Z boson to electrons to muons, corresponding to approximately 7% of its branching ratio. These final states provide  process. The template for the Z + jets component is derived from a control sample, obtained by inverting the photon identification criteria with respect to the signal region.
The m Zγ distribution in the analysis signal region and the fit to the spectrum is shown in Fig. 3. The observed (expected) significance with respect to a null hypothesis is 2.2σ (1.2σ). The best-fit value of the signal strength parameter μ Zγ obtained from the statistical analysis is 2.0 +1.0 −0.9 . The uncertainty of μ Zγ is dominated by the statistical error, which corresponds to ±0.9. The obtained limit is μ Zγ < 3.6 at the 95% confidence level.
Overall, the expected sensitivity improves by a factor 2.4 compared to the previous ATLAS publication using a dataset corresponding to 36 fb −1 [12]. Of this improvement, a factor of about two is due to the larger size of the dataset, and a factor of about 20% is due to the improvement in the analysis techniques.

SEARCH FOR RESONANCES DECAYING INTO PHOTON PAIRS
This analysis [13] searches for new resonances decaying into two photons in the diphoton mass region m γγ > 160 GeV. It assumes two benchmark BSM scenarios: a spin-0 resonant state, that arises from, e.g., extensions to the Higgs sector [14,15], and a spin-2 graviton, that arises from, e.g., Kaluza-Klein [16] excitations in Randall-Sundrum models [17,18]. For the spin-0 scenario, a ratio of the width of the resonance Γ X to its mass m X up to Γ X /m X = 10% is assumed. The upper limits are  The analysis first selects events with at least two isolated photons. The signal extraction is performed by a fit to the m γγ distribution in the analysis signal region. The signal component is modelled by a double-sided Crystal Ball function for the spin-0 state, and it is convolved with a relativistic Breit-Wigner form for signal scenarios with large width. The background component is modelled by an analytic function with free parameters. The validation of this background-modelling approach is performed using templates of the m γγ distribution. The template for the γγ background component is obtained from the simulated samples of the process. To suppress the impact of the statistical fluctuations of the simulated samples on the validation procedure, a smoothing is applied to the template using the functional decomposition method. The template for the γ + jets component is derived from the control sample of the collision data, obtained in a region with the inverted photon identification criteria with respect to the analysis signal region.
The fitted m γγ distribution in the signal region is shown in Fig. 4. No significant excess is observed. The most significant excess corresponds to a local significance of 3.29σ, for a resonance with a mass of 684 GeV and a narrow width (Γ X /m X ∼ 0 for the spin-0 scenario and the graviton coupling k/M Pl = 0.01 for the spin-2 scenario). This corresponds to a global significance of 1.30σ and 1.36σ for the two scenarios, respectively. The upper limits on the signal cross sections times branching ratios are shown in Fig. 5.

SUMMARY
Searches for unobserved processes in the Higgs sector provide probes for physics beyond the Standard Model. The results are presented for the searches for rare decays of the SM Higgs boson, and the search for a new diphoton resonance, performed by the ATLAS experiment at the LHC.
The H → μμ and H → Zγ processes are rare decays of the SM Higgs boson with branching ratios of O(10 −4 ). The searches for these rare processes have resulted in the observed significance at the level of 2σ. For the search for a new diphoton resonance, no significant excess is observed. The upper limits on the fiducial cross sections are evaluated as low as 0.03 fb for a resonance mass of 3 TeV. These results gain from improvements in the analysis sensitivity due to the increase in the size of the data set, and advancements in the analysis techniques.