Gross-Neveu-Yukawa SO(2) and SO(3) tensorial criticality

We investigate the relativistic SO(2)- and SO(3)-invariant Gross-Neveu-Yukawa field theories for real, rank-two, symmetric, traceless tensor order parameters coupled to Nf flavors of two-component Dirac fermions. These field theories arise as an effective description of fractionalized spin-orbital liquids. The two theories are the simplest and special cases of the more general class of field theories with SO(N) symmetric tensor order parameter coupled to Dirac fermions, in which the symmetry is low enough to allow only one and not the usual two quartic self-interaction terms. Using a two-loop renormalization group near the upper critical dimension, we demonstrate that the theory exhibits an exotic critical fixed point for N=3 and the concomitant continuous phase transition for any value of Nf. For N=2 the theory is equivalent to the chiral XY model. We discuss the crucial role of the symmetry-allowed sextic self-interactions in the selection of the ground state configuration in the case of SO(3). The universal quantities such as the anomalous dimensions of order parameters and fermions, the correlation length exponent, and the mass gap ratio between order parameter and fermion masses are computed up to ϵ2 order.