Two-particle pseudorapidity correlations are measured in $\sqrt{s_{\rm{NN}}}$ = 2.76 TeV Pb+Pb, $\sqrt{s_{\rm{NN}}}$ = 5.02 TeV $p$+Pb, and $\sqrt{s}$ = 13 TeV $pp$ collisions at the LHC, with total integrated luminosities of approximately 7 $\mu\mathrm{b}^{-1}$, 28 $\mathrm{nb}^{-1}$, and 65 $\mathrm{nb}^{-1}$, respectively. The correlation function $C_{\rm N}(\eta_1,\eta_2)$ is measured as a function of event multiplicity using charged particles in the pseudorapidity range $|\eta|<2.4$. The correlation function contains a significant short-range component, which is estimated and subtracted. After removal of the short-range component, the shape of the correlation function is described approximately by $1+\langle{a_1^2}\rangle \eta_1\eta_2$ in all collision systems over the full multiplicity range. The values of $\sqrt{\langle{a_1^2}\rangle}$ are consistent between the opposite-charge pairs and same-charge pairs, and for the three collision systems at similar multiplicity. The values of $\sqrt{\langle{a_1^2}\rangle}$ and the magnitude of the short-range component both follow a power-law dependence on the event multiplicity. The $\eta$ distribution of the short-range component, after symmetrizing the proton and lead directions in $p$+Pb collisions, is found to be smaller than that in $pp$ collisions with comparable multiplicity.