A measurement of the inclusive b-jet production cross section is presented in pp and p-Pb collisions at $\sqrt{s_{\mathrm{NN}}}~=0.5~$TeV, using data collected with the ALICE detector at the LHC. The jets were reconstructed in the central rapidity region $|\eta|<0.5$ from charged particles using the anti-$k_{\rm T}$ algorithm with resolution parameter $R=0.4$. Identification of b jets exploits the long lifetime of b-hadrons, using the properties of secondary vertices and impact parameter distributions. The $p_\mathrm{T}$-differential inclusive production cross section of b jets, as well as the corresponding inclusive b-jet fraction, are reported for pp and p-Pb collisions in the jet transverse momentum range $10 \le p_\mathrm{T,ch\,jet} \le 100$ GeV/$c$, together with the nuclear modification factor, $R_{\rm pPb}^\text{b-jet}$. The analysis thus extends the lower $p_\mathrm{T}$ limit of b-jet measurements at the LHC. The nuclear modification factor is found to be consistent with unity, indicating that the production of b jets in p-Pb at $\sqrt{s_{\mathrm{NN}}}~=0.5~$TeV is not affected by cold nuclear matter effects within the current precision. The measurements are well reproduced by POWHEG NLO pQCD calculations with PYTHIA fragmentation.