A search is performed for new phenomena in events with a high-transverse momentum photon and jets collected in 36.7/fb of pp collisions at a centre-of-mass energy of 13 TeV recorded with the ATLAS detector at the Large Hadron Collider. The invariant mass distribution of the leading photon and jet is examined to look for a resonant production of new particles or the presence of new high-mass states beyond the Standard Model. No significant deviation is observed in the mass distribution from the background-only hypothesis and cross section limits for generic Gaussian-shaped resonances are extracted. Excited quarks hypothesized in the quark compositeness model and high-mass states predicted in the quantum black hole models with extra dimensions are also examined in the analysis. The observed data exclude, at 95% confidence level, mass range below 5.3 TeV for the excited quarks and 7.1 TeV (4.4 TeV) for the quantum black holes with six (one) extra dimensions in the Arkani-Hamed-Dimopoulos-Dvali (Randall-Sundrum) model.