This paper presents measurements of charged-hadron spectra obtained in $pp$, $p$+Pb, and Pb+Pb collisions at $\sqrt{s}$ or $\sqrt{s_{_\text{NN}}}=5.02$ TeV, and in Xe+Xe collisions at $\sqrt{s_{_\text{NN}}}=5.44$ TeV. The data recorded by the ATLAS detector at the LHC have total integrated luminosities of 25 pb${}^{-1}$, 28 nb${}^{-1}$, 0.50 nb${}^{-1}$, and 3 $\mu$b${}^{-1}$, respectively. The nuclear modification factors $R_{p\text{Pb}}$ and $R_\text{AA}$ are obtained by comparing the spectra in heavy-ion and $pp$ collisions in a wide range of charged-particle transverse momenta and pseudorapidity. The nuclear modification factor $R_{p\text{Pb}}$ shows a moderate enhancement above unity with a maximum at $p_{\mathrm{T}} \approx 3$ GeV; the enhancement is stronger in the Pb-going direction. The nuclear modification factors in both Pb+Pb and Xe+Xe collisions feature a significant, centrality-dependent suppression. They show a similar distinct $p_{\mathrm{T}}$-dependence with a local maximum at $p_{\mathrm{T}} \approx 2$ GeV and a local minimum at $p_{\mathrm{T}} \approx 7$ GeV. This dependence is more distinguishable in more central collisions. No significant $|\eta|$-dependence is found. A comprehensive comparison with several theoretical predictions is also provided. They typically describe $R_\text{AA}$ better in central collisions and in the $p_{\mathrm{T}}$ range from about 10 to 100 GeV.