Measurement of cos2beta in B0->D(*)h0 Decays with a Time-Dependent Dalitz Plot Analysis of D->KSpi+pi-

We study the time-dependent Dalitz plot of D->KSpi+pi- in B0->D(*)h0 decays, where h0 is a pi0, eta, eta', or omega meson and D*->Dpi0 using a data sample of 383 X 10^6 Upsilon(4S)->BBbar decays collected with the BABAR detector. We determine cos2beta = 0.42+-0.49+-0.09+-0.13, sin2beta = 0.29+-0.34+-0.03+-0.05, and |lambda| = 1.01+-0.08+-0.02, where the first error is statistical, the second is the experimental systematic uncertainty, and the third, where given, is the Dalitz model uncertainty. Assuming the world average value for sin2beta and |lambda|=1, cos2beta>0 is preferred over cos2beta<0 at 86% confidence level.

the experimental systematic uncertainty, and the third, where given, is the Dalitz model uncertainty. Assuming the world average value for sin 2β and |λ| = 1, cos 2β > 0 is preferred over cos 2β < 0 at 86% confidence level.
PACS numbers: 13.25.Hw,12.15.Hh,11.30.Er Time-dependent CP asymmetries in B 0 meson decays, resulting from the interference between decays with and without B 0 -B 0 mixing, have been studied with high precision in b → ccs decay modes by the BABAR and Belle collaborations [1]. These studies measure the asymmetry amplitude sin 2β, where β = −arg(V cd V * cb /V td V * tb ) is a phase in the Cabibbo-Kobayashi-Maskawa (CKM) quark-mixing matrix [2], V . The CP violating phase 2β, inferred from sin 2β, has a two-fold ambiguity, 2β and π − 2β (four-fold ambiguity in β). This ambiguity can be resolved by studying decay modes that involve multibody final states B 0 → J/ψK 0 S π 0 [3], D[K 0 S π + π − ]h 0 [4], D * + D * − K 0 S [5] or K + K − K 0 [6], where the knowledge of the variation of the strong phase differences as a function of phase space allows one also to measure cos 2β.
In this Letter, we present a study of CP asymmetry in B 0 → D ( * ) h 0 [7] decays with a time-dependent Dalitz plot analysis of D → K 0 S π + π − [8], where h 0 is a π 0 , η, η ′ , or ω meson. The B 0 → D ( * ) h 0 decay is dominated by a color-suppressed b → cud tree amplitude. The diagram b → uḑ, which involves a different weak phase, is suppressed by Neglecting the suppressed amplitude, we factorize the decay amplitude of the chain The D 0 and D 0 decay amplitudes are functions of the Dalitz plot variables In the Υ (4S) → B 0 B 0 system, the rate of a neutral B meson decaying at proper decay time t rec , the other B (B tag ) at t tag , and the D decaying at a point on the Dalitz plot, is proportional to where the upper (lower) sign is for events with B tag decaying as a B 0 (B 0 ), ∆t = t rec −t tag , Γ is the decay rate of the neutral B meson, λ = e −2iβ (A B /A B ), ∆m is the B 0 -B 0 mixing frequency, ξ h 0 is the CP eigenvalue of h 0 , and (−1) L is the orbital angular momentum factor. Here we have assumed CP -conservation in mixing and neglected decay width differences. For Dh 0 modes, including factors from D * decay [9]. In the last term of expression 1 we can rewrite and treat cos 2β and sin 2β as independent parameters. We fully reconstruct B 0 → D ( * ) h 0 candidates from a data sample of (383±4)×10 6 Υ (4S) decays into BB pairs collected with the BABAR detector at the asymmetricenergy e + e − PEP-II collider. The BABAR detector is described in detail elsewhere [10]. The decay modes used are Dπ 0 , Dη, Dη ′ , Dω, D * π 0 , and D * η, with D * → Dπ 0 , D → K 0 S π + π − , K 0 S → π + π − , π 0 → γγ, η → γγ, π 0 π + π − , η ′ → η π + π − , and ω → π 0 π + π − .
Charged tracks are considered to be pions. The K 0 S candidate is reconstructed from π + π − pairs, whose χ 2 probability of forming a common vertex is greater than 0.1%, with invariant mass within 10 MeV/c 2 of the nominal K 0 S mass [11]. The distance between the K 0 S decay vertex and the primary interaction point projected on the x-y plane (perpendicular to the beam axis) is required to be greater than three times its measurement uncertainty. The angle θ K between the K 0 S momentum and the line connecting the production and decay vertices of the K 0 S on the x-y plane is required to satisfy cos θ K > 0.992.
An energy cluster in the electromagnetic calorimeter, isolated from any charged tracks and with the expected lateral shower shape for photons, is considered a photon candidate. A pair of photons forms a π 0 → γγ (η → γγ) candidate if both photon energies exceed 30 (100) MeV and the invariant mass of the pair is between 100 and 160 MeV/c 2 (508 and 588 MeV/c 2 ). If the η is paired with a D * , the invariant mass window is tightened to 515 < m γγ < 581 MeV/c 2 . The η → γγ candidate is rejected if either photon, when combined with any other photon in the event, has an invariant mass within 6 MeV/c 2 of the nominal π 0 mass. We perform a kinematic fit to the photon pair with its invariant mass constrained at the nominal π 0 or η mass and reject candidates with a fit probability less than 0.1%.
The η/ω → π 0 π + π − , η ′ → η π + π − , and D → K 0 S π + π − candidates are formed by combining a π 0 , η, or K 0 S with two charged pions. The χ 2 probability of the decay products coming from a common vertex for h 0 (D) is required to be greater than 0.1% (1%). The momentum of the π 0 and the η candidates used in ω and η ′ reconstruction must be greater than 200 MeV/c. The invariant masses of the η, η ′ , and ω candidates are required to be within 10, 8 and 18 MeV/c 2 of their respective nominal masses, which correspond to approximately twice the RMS of the signal distributions. We retain D candidates within 60 MeV/c 2 of the nominal D 0 mass, approximately 10 times its mass resolution, to include sufficient data in the sideband. A kinematic fit is performed on the D candidate to constrain its mass to the nominal D 0 mass. A D * → Dπ 0 candidate is accepted if the invariant mass difference between D * and D candidates is within 3 MeV/c 2 of the nominal mass difference.
The signal is characterized by the kinematic variables where the asterisk denotes the quantities evaluated in the center-of-mass (c.m.) frame, the subscripts 0, beam and B denote the e + e − system, the beam and the B candidate, respectively, and √ s is the c.m. energy. For signal events, m ES peaks near the B 0 mass with a resolution of about 3 MeV/c 2 , and ∆E peaks near zero, with a resolution that varies by mode. We require m ES > 5.23 GeV/c 2 and select events with |∆E| < 80 MeV for modes with π 0 , η → γγ, and |∆E| < 40 MeV for modes with η, ω → π 0 π + π − , or η ′ → η π + π − .
The proper decay time difference ∆t is determined from the measured distance between the two B decay vertices projected onto the boost axis and the boost (βγ = 0.56) of the c.m. system. The reconstructed |∆t| and its uncertainty σ ∆t are required to satisfy |∆t| < 15 ps and σ ∆t < 2.5 ps. The flavor of B tag is identified from particles that do not belong to the reconstructed B meson using a neural network based flavor-tagging algorithm [12].
The main background is from the continuum e + e − → qq (q = u, d, s, c). We use a Fisher discriminant (F ) to separate the more isotropic BB events from more jet-like qq events [13]. The requirement on F is optimized with simulation. Another major background for the D * π 0 mode comes from color-allowed B − → D 0 ρ − (ρ − → π 0 π − ) decays, which mimic signal if the π − is missed from reconstruction while a random π 0 is included. We veto the B 0 candidate if the combination of another charged pion in the event with the D and the π 0 in the B 0 candidate is consistent with a charged B decay. In total we select 4450 events, of which 2843 events have useful tagging information (tagged).
The signal and background yields are determined by a fit to the (m ES , m D ) distributions using a twodimensional probability density function (PDF), where m D denotes the K 0 S π + π − invariant mass. We divide the sample into four categories to take into account different background levels: (1) Dπ 0 , (2) Dη and Dη ′ (3) Dω, and (4) D * h 0 . The PDF has five components: (a) signal, and backgrounds that peak in (b) both m ES and m D , (c) m ES but not m D , (d) m D but not m ES , and (e) neither distribution. Both peaks are modeled by a Crystal Ball line shape [14]. The non-peaking component is modeled by a straight line in m D and a threshold function [15] in m ES . We fit the four categories of events simultaneously, allowing the m ES peak shape to be different but letting them share the m D shape and m ES background parameters. We first determine the amount of the peaking component (b) from simulated events and then fit to data allowing all other components to vary. We obtain 463 ± 39 signal events (335 ± 32 tagged). The contribution from each mode is shown in Table I. The m ES and m D distributions are shown in Fig. 1.
The D 0 → K 0 S π + π − Dalitz plot has been studied in detail [16,17]. We use the isobar formalism described in [18] to express A D 0 as a sum of two-body decay matrix elements (A r ) and a non-resonant (NR) contribution, The function A r (m 2 + , m 2 − ) is the Lorentz-invariant expression for the matrix element of a D 0 decaying into K 0 S π + π − through an intermediate resonance r, parameterized as a function of the position on the Dalitz plot. The resonances in the model include K * (892), K * 0 (1430), K * 2 (1430), K * (1410) and K * (1680) for both K 0 S π + and K 0 S π − , and ρ (770), ω(782), f 0 (980), f 0 (1370), f 2 (1270), ρ(1450), and two scalar terms σ and σ ′ in the π + π − system. Details of the Dalitz model and the parameters (determined from data) can be found in [17].
To perform the time-dependent Dalitz plot analysis, we expand the PDF to include ∆t and Dalitz plot dependence. The signal component is proportional to expression 1, modified to account for the probability of mis-identifying the B tag flavor (mistag), and is convolved with a sum of three Gaussian distributions [19]. The mistag parameters and the resolution function are determined from a large data control sample of B 0 → D ( * )− h + decays, where h + is a π + , ρ + , or a + 1 meson. Each of the background components consists of a product of ∆t and (m 2 + , m 2 − ) PDFs. The components that peak in m D use A D 0 (m 2 + , m 2 − ) as their Dalitz model. The model for components that are flat in m D is an incoherent sum of a phase space contribution and several resonances. The choice of resonances and their relative contributions are determined empirically from events outside the m D peak. The ∆t model for components that peak in m ES is a simple exponential decay convolved with the resolution function used in the signal component. For the nonpeaking background, we use a zero-lifetime component convolved with a double-Gaussian resolution function for events with a real D because they are dominated by cc events, and we add an exponential decay component for events without a real D to account for B background. We fit the m ES , m D , and ∆t distributions, with m ES and m D shapes and background fractions fixed by the previous fit for event yields, to determine the ∆t parameters for backgrounds. We then perform the final fit adding Dalitz plot variables to determine cos 2β, sin 2β and |λ|. Table I shows the nominal fit result (All) and the results of a fit allowing cos 2β and sin 2β to be different among the four types of events. The correlations are ρ(cos 2β, sin 2β) = 2%, ρ(|λ|, cos 2β) = 2%, and ρ(|λ|, sin 2β) = −2%. The Dalitz plot projections are shown in Fig. 2. Figure 3 shows the time-dependent asymmetries (N + − N − )/(N + + N − ), where N + (N − ) is the number of B 0 (B 0 ) tagged events, for events in various Dalitz plot regions. Events in the D → K 0 S ρ region are dominated by a single CP eigenstate, thus the asymmetry is proportional to sin 2β sin(∆m∆t). Events near D → K * ± π ∓ are dominated by decays to a definite flavor, and therefore exhibit a cos(∆m∆t) behavior. The dominant systematic uncertainty is the Dalitz plot model dependence. The Dalitz model includes scalar terms σ and σ ′ , which are not well established, in order to achieve a good quality fit [17]. We study the effect of these two scalars by simulating a number of datasets, each of which is 50 times the size of the data, according to the PDF, and repeat the final fit using both the nominal PDF and the PDF without the two scalars. We Time-dependent asymmetries for (a) D → K 0 S ρ region (|m π + π − − 770| < 150), where the opposite CP asymmetry in D * h 0 has been taken into account, (b) D → K * + π − region, and (c) D → K * − π + region (|m K 0 S π − 892| < 50). Units are MeV/c 2 . Curves are projections of the PDF.
compare the results between the two fits in each dataset and conservatively take the quadratic sum of the mean and RMS of the differences as the systematic uncertainty: σ(cos 2β) = 0.13, σ(sin 2β) = 0.05, and σ(|λ|) < 0.01. Many parameters are pre-determined in fits to control samples and to data without the Dalitz variables. We randomize them according to a Gaussian distribution whose width equals one standard deviation of each parameter, taking correlations into account, and repeat the final fit. The width of the distribution is taken as the systematic uncertainty: σ(cos 2β) = 0.06, σ(sin 2β) = 0.02 from Dalitz model parameters; σ(cos 2β) = 0.05, σ(sin 2β) = 0.02 from m D and m ES shape parameters; σ(cos 2β) < ∼ 0.01, σ(sin 2β) < ∼ 0.01 from background ∆t parameters, tagging parameters, or signal ∆t resolution function. We also vary the peaking background fractions by the statistical uncertainty found in simulation and find the variations are σ(cos 2β) = 0.02 and σ(sin 2β) = 0.01. Other sources of uncertainty such as B 0 -B 0 mixing frequency, B lifetimes, background Dalitz model and reconstruction efficiency variation over the Dalitz plot are negligible. In all cases, the uncertainty on |λ| is less than 0.01. The only significant uncertainty on |λ| (∼ 0.02) is from the interference between the CKM-suppressed b → ucd and CKM-favored b → cud amplitudes in some B tag final states [20]. This effect is studied with simulation. Summing over all contributions in quadrature, we obtain total experimental systematic uncertainties σ(cos 2β) = 0.09, σ(sin 2β) = 0.03, and σ(|λ|) = 0.02.
To resolve the ambiguity in 2β, we generate two sets of toy simulation samples, one with cos 2β = 1 − S 2 0 ≡ C 0 , and the other with cos 2β = −C 0 , where S 0 = 0.678, the world average of sin 2β [21], and fit each sample while fixing sin 2β = S 0 and |λ| = 1. For data, this configuration results in cos 2β = 0.43 ± 0.47. We then use double-Gaussian functions, h ± (x) for ±C 0 hypotheses, to model the probability density of the resulting cos 2β distributions, smeared by the experimental systematic uncertainty and the uncertainty of C 0 . The confidence level (C.L.) of preferring cos 2β = +C 0 over −C 0 is defined as h + (x)/[h + (x) + h − (x)] if cos 2β = x is observed