A search for long-lived particles decaying into hadrons and at least one muon is presented. The analysis selects events that pass a muon or missing-transverse-momentum trigger and contain a displaced muon track and a displaced vertex. The analyzed dataset of proton-proton collisions at $\sqrt{s} = 13$ TeV was collected with the ATLAS detector and corresponds to 136 fb$^{-1}$. The search employs dedicated reconstruction techniques that significantly increase the sensitivity to long-lived particle decays that occur in the ATLAS inner detector. Background estimates for Standard Model processes and instrumental effects are extracted from data. The observed event yields are compatible with those expected from background processes. The results are presented as limits at 95% confidence level on model-independent cross sections for processes beyond the Standard Model, and interpreted as exclusion limits in scenarios with pair-production of long-lived top squarks that decay via a small $R$-parity-violating coupling into a quark and a muon. Top squarks with masses up to 1.7 TeV are excluded for a lifetime of 0.1 ns, and masses below 1.3 TeV are excluded for lifetimes between 0.01 ns and 30 ns.