Decays of the 125 GeV Higgs boson into a Z boson and a $\rho^0$(770) or $\phi$(1020) meson are searched for using proton-proton collision data collected by the CMS experiment at the LHC at $\sqrt{s} = $ 13 TeV. The analysed data set corresponds to an integrated luminosity of 137 fb$^{-1}$. Events are selected in which the Z boson decays into a pair of electrons or a pair of muons, and the $\rho$ and $\phi$ mesons decay into pairs of pions and kaons, respectively. No significant excess above the background model is observed. As different polarization states are possible for the decay products of the Z boson and $\rho$ or $\phi$ mesons, affecting the signal acceptance, scenarios in which the decays are longitudinally or transversely polarized are considered. Upper limits at the 95% confidence level on the Higgs boson branching fractions into Z$\rho$ and Z$\phi$ are determined to be 1.04-1.31% and 0.31-0.40%, respectively, where the ranges reflect the considered polarization scenarios; these values are 740-940 and 730-950 times larger than the respective standard model expectations. These results constitute the first experimental limits on the two decay channels.