Parton energy loss in the quark-gluon plasma (QGP) is studied with a measurement of photon-tagged jet production in 1.7 nb$^{-1}$ of Pb+Pb data and 260 pb$^{-1}$ of $pp$ data, both at $\sqrt{s_\mathrm{NN}} = 5.02$ TeV, with the ATLAS detector. The process $pp \to \gamma$+jet+$X$ and its analogue in Pb+Pb collisions is measured in events containing an isolated photon with transverse momentum ($p_\mathrm{T}$) above $50$ GeV and reported as a function of jet $p_\mathrm{T}$. This selection results in a sample of jets with a steeply falling $p_\mathrm{T}$ distribution that are mostly initiated by the showering of quarks. The $pp$ and Pb+Pb measurements are used to report the nuclear modification factor, $R_\mathrm{AA}$, and the fractional energy loss, $S_\mathrm{loss}$, for photon-tagged jets. In addition, the results are compared with the analogous ones for inclusive jets, which have a significantly smaller quark-initiated fraction. The $R_\mathrm{AA}$ and $S_\mathrm{loss}$ values are found to be significantly different between those for photon-tagged jets and inclusive jets, demonstrating that energy loss in the QGP is sensitive to the colour-charge of the initiating parton. The results are also compared with a variety of theoretical models of colour-charge-dependent energy loss.