A first search is presented for vector-like leptons (VLLs) decaying into a light long-lived pseudoscalar boson and a standard model $\tau$ lepton. The pseudoscalar boson is assumed to have a mass of 2 GeV and to decay exclusively into a pair of photons. It is identified using the CMS muon system. The analysis is carried out using a data set of proton-proton collisions at a center-of-mass energy of 13 TeV collected by the CMS experiment in 2016-2018, corresponding to an integrated luminosity of 138 fb$^{-1}$. Selected events contain at least one pseudoscalar boson decaying electromagnetically in the muon system and at least one hadronically decaying $\tau$ lepton. No significant excess of data events is observed compared to the background expectation. Upper limits are set at 95% confidence level on the vector-like lepton production cross section as a function of the VLL mass and the pseudoscalar boson mean proper decay length. The observed and expected exclusion ranges of the VLL mass extend up to 700 and 670 GeV, respectively, depending on the pseudoscalar boson lifetime.