A search for $Z$ boson decays into a $\tau$ lepton and another lepton of different flavour ($\ell$) with opposite electric charge is presented. The data analysed correspond to $139\text{ fb}^{-1}$ of proton-proton collisions at a centre-of-mass energy of $\sqrt{s}=13\text{ TeV}$ and $20.3\text{ fb}^{-1}$ at $\sqrt{s}=8\text{ TeV}$, recorded with the ATLAS detector at the LHC. Events are categorised according to the multiplicities and kinematics of reconstructed electrons, muons and $\tau$ leptons, as well as the missing transverse momentum. The signal region (SR) is defined by a $e^\pm\tau^\mp$ or $\mu^\pm\tau^\mp$ final state with $m_\mathrm{T}(\tau_\mathrm{had-vis}, E_\mathrm{T}^\mathrm{miss})<35\text{ GeV}$. The same-sign validation region (VRSS) is defined similarly to the SR but with a $e^\pm\tau^\pm$ or $\mu^\pm\tau^\pm$ final state. The control region enhanced in $Z\rightarrow\tau\tau$ events (CRZ$\tau\tau$) is defined similarly to the SR but with $m_\mathrm{T}(\tau_\mathrm{had-vis}, E_\mathrm{T}^\mathrm{miss})>35\text{ GeV}$, $m_\mathrm{T}(\ell, E_\mathrm{T}^\mathrm{miss})<40$ GeV and $70\text{ GeV}<m_\mathrm{coll}(\ell,\tau)<110\text{ GeV}$, where $m_\mathrm{coll}$ is the collinear mass. Neural network (NN) classifiers are used to discriminate between signal and background events. The combined NN output is fit to data in the SR and CRZ$\tau\tau$ to derive exclusion upper limits on the lepton-flavour-violating branching fraction $\mathcal{B}(Z\rightarrow\ell\tau)$.