Measurements of the differential jet cross section are presented as a functionof jet mass in dijet events, in bins of jet transverse momentum, with and withouta jet grooming algorithm. The data have been recorded by the CMS Collaborationin proton-proton collisions at the LHC at a center-of-mass energy of 13\TeV andcorrespond to an integrated luminosity of 2.3\fbinv. The absolute cross sections show slightly different jettransverse momentum spectra in data and Monte Carlo event generatorsfor the settings used. Removing this transverse momentum dependence,the normalized cross section for ungroomed jets is consistent with theprediction from Monte Carlo event generators for masses below 30\% ofthe transverse momentum. The normalized cross section for groomed jetsis measured with higher precision than the ungroomed crosssection. Semi-analytical calculations of the jet mass beyond leadinglogarithmic accuracy are compared to data, as well as predictions atleading order and next-to-leading order, which include partonshowering and hadronization. Overall, in the normalized cross section,the theoretical predictions agree with the measured cross sectionswithin the uncertainties for masses from 10 to 30\% of the jettransverse momentum.