Measurements of azimuthal angle and transverse momentum ($p_\mathrm{T}$) correlations of isolated photons and associated jets are reported for pp and PbPb collisions at $\sqrt{s_{_{\mathrm{NN}}}} =$ 5.02 TeV. The data were recorded with the CMS detector at the CERN LHC. For events containing a leading isolated photon with $p_\mathrm{T}^\gamma >$ 40 GeV$/c$ and an associated jet with $p_\mathrm{T}^\text{jet} >$ 30 GeV$/c$, the photon+jet azimuthal correlation and $p_\mathrm{T}$ imbalance in PbPb collisions are studied as functions of collision centrality and $p_\mathrm{T}^\gamma$. The results are compared to pp reference data collected at the same collision energy and to predictions from several theoretical models for parton energy loss. No evidence of broadening of the photon+jet azimuthal correlations is observed, while the ratio $p_\mathrm{T}^\text{jet}/p_\mathrm{T}^\gamma$ decreases significantly for PbPb data relative to the pp reference. All models considered agree within uncertainties with the data. The number of associated jets per photon with $p_\mathrm{T}^\gamma >$ 80 GeV$/c$ is observed to be shifted towards lower $p_\mathrm{T}^\text{jet}$ values in central PbPb collisions compared to pp collisions.